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March 13, 2001 


MEMORANDUM FOR DIRECTOR, DEFENSE PROCUREMENT 

SUBJECT: Audit Report on Standard Procurement System Use and User Satisfaction 
(Report No. D-2001-075) 


We are providing this report for review and comment. We conducted the audit 
in response to a congressional request. We considered comments from the Director, 
Defense Procurement, and the Defense Contract Management Agency when preparing 
the final report. 

DoD Directive 7650.3 requires that all management comments be resolved 
promptly. The Director, Defense Procurement, and Defense Contract Management 
Agency comments were partially responsive. As a result of management comments, we 
added Recommendation A.l.b.4.; revised Recommendations B.l.a and B.l.c., deleted 
Recommendation B.2.b; and revised and redirected Recommendation B.2. Therefore, 
we request the Director, Defense Procurement, provide additional comments on 
Recommendations A.I., and B.l. through B.3., in response to the fmal report. We 
request management provide comments by May 14, 2001. 

We appreciate the courtesies extended to the audit staff. Questions on the audit 
should be directed to Ms. Kimberley A. Caprio at (703) 604-9139 (DSN 664-9139) 
(kcaprio@dodig.osd.mil) or Ms. Cynthia G. Williams at (703) 604-9168 
(DSN 664-9168) (cwilliams@dodig.osd.mil). See Appendix D for the report 
distribution. The audit team members are listed inside the back cover. 

JbcuMLft. 

David K. Steensma 
Acting Assistant Inspector General 
for Auditing 
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(Project No. D2000FG-0091) 

Standard Procurement System Use and User Satisfaction 


Executive Summary 


Introduction. The audit was performed in response to concerns expressed by the 
Chairman, House of Representatives Committee on Budget that DoD was not effectively 
spending Federal funds to acquire the Standard Procurement System (SPS) and SPS lacked 
needed functionality. SPS is an automated information system that will support 
procurement functions from receipt of requirements until contract closeout at all DoD 
procurement organizations. SPS is intended to replace 76 procurement systems and 
manual processes. As of December 30, 2000, the Program Management Office reported 
that SPS was used by 16,207 users at 745 DoD sites. By the end of FY 2003, SPS is 
expected to serve 43,000 users at 1,100 DoD sites. Estimated costs for SPS are 
$433.5 million to procure commercial software licenses and support services. Estimated 
life-cycle costs for FY 1995 through FY 2005 are $3.7 billion. Operational benefits from 
SPS are estimated at $1.4 billion derived primarily from increased productivity and 
reduced costs associated with paper transactions. 

The Director, Defense Procurement, selected a contractor to provide a commercial off- 
the-shelf product to accomplish 45 percent of a total of 299 DoD procurement functions. 
The remaining 55 percent would be accomplished through modifications to the 
commercial product. As of December 2000, DoD had deployed four versions of SPS, 
through version 4.1. 

Objectives. The audit objective was to evaluate allegations related to SPS functionality, 
user satisfaction, system implementation and operation, and system controls. An additional 
objective was to follow up on recommendations made in our May 1999 report. System 
controls will be discussed in a future report. 

Results. Audit results were based on responses to a web-based survey of statistically 
selected personnel from a population of SPS 4.1 users at 534 DoD procurement sites (see 
Appendix C). About 85.9 percent of SPS users stated that SPS was available always or 
most of the time. The SPS Program Management Office in the Defense Contract 
Management Agency had taken steps to better meet user needs, and respondents stated that 
SPS had the potential of being a very effective and useful tool, but more needed to be done 
to improve the software and gain greater acceptance and user confidence. Specifically, the 
projected survey results indicated that: 

• 60.8 percent of SPS users preferred a procurement system other than SPS, 

• 45.8 percent of SPS users stated that the number of workarounds increased, 

• 51.4 percent of SPS users stated that productivity has not increased since SPS 
version 4.1 was implemented, and 

• 63.5 percent of SPS users stated that SPS had not substantially contributed to the 
DoD goal of paperless contracting (finding A). 



Further, based on survey responses, we projected that about 26.5 percent of the personnel 
licensed to use SPS version 4.1 have not used it because SPS either lacked the functionality 
for those sites or employees received SPS when it was not needed to perform their jobs. 

We estimate that the Program Management Office spent up to $2.1 million of the 
$7.9 million in license costs on licenses for users who could not or would not use SPS 
(finding B). 

DoD has experienced a 50 percent reduction in the procurement workforce without a 
commensurate reduction in workload. Conceptually, SPS should assist in automating and 
standardizing a variety of procurement tasks and thus assist in more efficiently completing 
the workload. According to the survey, however, functionality remains a serious concern. 
Management needs to respond to this concern when deploying new SPS versions and, if 
SPS does not fully meet mission needs, should consider supplementary and alternative tools 
for the procurement workforce. 

Summary of Recommendations. We recommend that the Director, Defense 
Procurement, direct the SPS Program Manager to perform tests to ensure that SPS meets 
user needs; to develop and evaluate SPS against quantifiable performance measures that 
gauge meeting mission objectives, improving productivity, achieving the goal of paperless 
contracting, and delivering intended benefits; and purchase future licenses only after sites 
clearly demonstrate the need and determine the number required. We also recommend that 
the Director, Defense Procurement, direct the DoD Components to coordinate training and 
the transition to the SPS, demonstrate that SPS meets site needs before deployment, and 
provide assurance that they have accurately determined license needs. 

Management Comments. The Director, Defense Procurement, and the Director, Defense 
Contract Management Agency, generally concurred with the recommendations that testing 
and performance measures are necessary, but stated that both already exist. The Director, 
Defense Procurement, also agreed to better coordinate training needs. Both Directors 
partially concurred that, prior to any future deployments of SPS, the DoD Components 
should determine that the version meets functionality requirements and identify the number 
of licenses required. However, both Directors disagreed that the DoD Components’ 
determinations need review or validation before future purchases of SPS licenses, stating 
that it is the responsibility of the DoD Components to determine license requirements. A 
discussion of management comments is in the Findings section of the report and the 
complete text is in the Management Comment section. 

Audit Response. The management comments were responsive regarding coordinating 
training needs. However, the other comments did not fully address the core issues 
identified by the customer survey. There is a need for more appropriate testing prior to 
future deployment. About 38 percent of respondents contend that SPS version 4.1 had only 
some or none of the functionality needed, despite testing. Present performance measures 
do not address mission needs such as enhancing customer service, reducing problem 
disbursements, increasing contractor personnel productivity, or eliminating redundancy. 

We agree that DoD Components have the responsibility for identifying SPS licensing needs 
with due diligence and we modified the related recommendation to allow management 
flexibility in determining how to obtain more credible assurance that stated requirements 
are valid. We request that the Director, Defense Procurement, provide additional 
comments in response to the final report by May 14, 2001. 
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Background 

The audit was performed in response to concerns expressed by the Chairman, 
House of Representatives Committee on Budget, that DoD was not effectively 
spending Federal funds to acquire the Standard Procurement System (SPS) and 
SPS lacked needed functionality. The report is one in a series and discusses 
functionality, user satisfaction, system implementation, and operation of 
the SPS. 

The SPS Program. The Director, Defense Procurement (DDP), initiated the 
SPS program in November 1994 to provide an automated system that would 
perform DoD procurement functions. The DDP is responsible for acquiring and 
deploying SPS, as well as for software installation, training, and all steps 
necessary to gain user acceptance of SPS. Procurement functions begin with 
receipt of a requirement and end with contract closeout. Standard procurement 
functions include, but are not limited to, acquiring supplies and services by 
describing requirements; determining the appropriate acquisition method; 
soliciting and selecting sources; and awarding, reporting, modifying, 
terminating, and closing out contracts. 

According to the Mission Need Statement, dated April 9, 1998, SPS should 
replace 76 legacy systems. In addition, SPS should provide standard policies, 
processes, procedures, shareable data, and electronic commerce 1 capability. 

SPS deployment should also provide more timely response to customer 
requirements, permit more cost-effective procurements, improve visibility of 
contract deliverables, reduce procurement lead times, reduce problem 
disbursements, and provide more accurate procurement information through 
shared data. Although the elimination of paper handling tasks was an expected 
benefit of SPS, the paperless contracting was not specifically addressed in the 
Mission Need Statement until April 8, 1998, when it was added in response to 
the DoD paperless contracting initiative. 

As of December 30, 2000, the Program Management Office (PMO) reported 
that SPS was used by 16,207 users at 745 sites. SPS was expected to serve 
43,000 users at 1,100 DoD procurement sites by the end of FY 2003. Program 
funding for SPS was estimated at $433.5 million to procure commercial licenses 
and support services for the software application. For FYs 1995 through 2005, 
estimated life-cycle and program costs were estimated at $3.7 billion with 
approximately $1.4 billion in operational benefits to be derived primarily from 
increased productivity and reduced costs associated with paper transactions. 

Responsibility for the SPS Program. The DDP, in the Office of the Under 
Secretary of Defense (Acquisition, Technology, and Logistics), had primary 
responsibility for the SPS program. On April 7, 1997, the DDP announced the 
selection of American Management Systems (AMS), Incorporated, Fairfax, 
Virginia, to furnish the procurement software and related services for SPS. The 
DDP delegated responsibility for managing and deploying SPS to the PMO 


'Electronic commerce is the interchange and processing of information using electronic 
techniques for accomplishing transactions. 
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within the Defense Contract Management Agency. The PMO also monitors 
contractor performance. Offices in each DoD Component responsible for SPS 
implementation acted as liaisons between SPS user organizations in the DoD 
Components and the PMO and established Component-level guidance on 
configuration control, data migration, interface development, training, site 
migration, and transition from legacy systems to SPS. 

Contract Details. DoD acquired SPS, a commercial off-the-shelf software 
application, under an indefinite-delivery contract with AMS. The contract 
required AMS to provide DoD with Procurement Desktop - Defense, a 
modified version of the AMS Procurement Desktop commercial computer 
software 2 that was also available to Federal agencies from the General Services 
Administration Supply Schedule. The contract required AMS to obtain and 
deploy the commercially available software, as well as provide related software 
support and support services, with options for continued maintenance, training, 
and support for up to 10 years. Because the initial commercial software would 
accomplish only 45 percent of DoD procurement functions, the contract 
required development of software enhancements and modifications to meet the 
remaining DoD functional requirements. Inspector General, DoD, Report 
No. 99-166, “Initial Implementation of the Standard Procurement System,” 

May 26, 1999, stated that the acquisition of SPS as a commercial product 
was questionable because of the need to make major modifications to the 
commercial software. 

Incremental Deployment. According to the SPS acquisition strategy, SPS 
would be delivered in increments of increasing functionality. SPS has been 
deployed sequentially with the Navy receiving SPS first, followed by the Army, 
Air Force, the Marine Corps, Defense Logistics Service Center, then other 
Defense agencies and the Defense Contract Management Agency (DCMA). To 
date, versions 3.1, 3.5, 4.0, and 4.1 have been deployed. The functionality of 
version 4.1 was designed for procurement functions at DoD posts, camps, and 
stations. Version 4.1 had five maintenance releases from 4. la through 4. le. 
Deployment of version 4.2 is scheduled in the second quarter of FY 2001. 
Version 4.2 will be designed for improved post, camp, and station functionality 
and for contract administration procurement functions. Deployment of 
version 5.0 is scheduled in the second quarter of FY 2002. The functionality of 
version 5.0 will be designed for major weapons system procurement functions. 
Deployment of version 5.1 is scheduled in the second quarter of FY 2003. The 
functionality of version 5.1 will be designed for procurement functions at 
inventory control points. 

Workforce Reduction Impacts. The Inspector General, DoD, issued Report 
No. D-2000-088, “DoD Acquisition Workforce Reduction Trends and 
Impacts,” February 29, 2000, on the impact of the workforce reduction. The 
report states that although acquisition organizations improved efficiency in 
contracting through acquisition reform initiatives, concern is warranted because 
staffing reductions have clearly outpaced productivity increases and acquisition 


Commercial computer software is a software developed or regularly used for non-governmental 
purposes that has been sold, leased, or licensed to the public; has been offered to the public; or 
will be available for commercial sale, lease, or license. 
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workforce capacity to handle its still formidable workload. There was cause for 
serious concern in the likelihood of the DoD acquisition workforce losing about 
55,000 experienced personnel through attrition by FY 2005 and in the overall 
disconnects between workload forecasts, performance measures, productivity 
indicators, and plans for workforce sizing and training. In addition, from 
FY 1990 through FY 2000 the number of procurement actions increased from 
about 13.2 million to about 15.5 million, or about 17 percent. The acquisition 
workforce reductions caused an increased backlog in closing out completed 
contracts, resulted in insufficient staff to manage requirements, reduced scrutiny 
and timeliness in reviewing acquisition actions, and increased procurement 
action lead time. 


Program Risk Previously Identified 

We previously issued two reports on the SPS implementation. Inspector 
General, DoD, Report No. 99-166 states that the SPS evolutionary approach did 
not provide some critical functional requirements to meet user needs and may 
not meet the mission need to standardize procurement policies, processes, and 
procedures. In addition, users were not receiving adequate training, guidance, 
and support from the contractor help desk. Inspector General, DoD, Report 
No. 96-219, “Allegations to the Defense Hotline Concerning the Standard 
Procurement System,” September 5, 1996, stated that the acquisition strategy 
increased the risks that the program would not meet the overall objective of a 
fully functional, DoD-wide standard procurement system and that user needs 
might not be met. 


Objectives 

The audit objective was to evaluate allegations related to SPS. Specific 
objectives were to evaluate SPS functionality, user satisfaction, system 
implementation, and operation. An additional objective was to follow up on 
recommendations made in our May 1999 report. The status of agreed-upon 
actions on recommendations in the May 1999 report is partially addressed in this 
report and may also be addressed in a future report. System controls will be 
discussed in a future report. See Appendix A for a discussion of the scope 
and methodology. 


Survey of SPS Users 

We developed a password protected web-based survey to gather data from 
statistically selected users regarding their SPS experience, training, and 
guidance. We statistically selected users who had SPS installed for at least 
3 months before completing the survey. See Appendix A for a full explanation 
of the statistical sampling methodology. See Appendix B for complete 
information on the statistical projections and Appendix C for complete 
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information on estimating the number of users of SPS version 4.1. The 
information contained in this report reflects the survey respondents’ views and 
opinions about SPS. 
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A. Standard Procurement System 
User Satisfaction 

Based upon responses to a statistical web-based survey of SPS 4.1 users 
at 534 DoD procurement sites, except for positive perceptions on SPS 
reliability, the audit generally substantiated that the user community 
remained fundamentally dissatisfied with SPS. The audit also confirmed 
congressional concerns regarding SPS functionality and cost. Survey 
respondents who stated that they used SPS version 4.1 indicated that 
more needed to be done to improve user satisfaction in the areas of SPS 
functionality, implementation, and operation. Specifically, user 
satisfaction needed to be improved because the PMO prematurely 
deployed SPS and had not developed performance measures to track 
whether SPS met the mission objectives and delivered intended benefits. 
In addition, DoD Components 3 did not effectively coordinate training 
and transition to SPS. As a result, the projected survey results indicated 
that about: 


• 60.8 percent of SPS users preferred a procurement system 
other than SPS, 

• 45.8 percent of SPS users stated that workarounds increased, 

• 51.4 percent of SPS users stated that productivity does not 
exceed productivity before SPS version 4.1, and 

• 63.5 percent of SPS users stated that SPS had not substantially 
contributed to the DoD goal of paperless contracting. 


Mission Requirements 

According to the SPS Mission Need Statement, April 9, 1998, the need for a 
standard DoD procurement system arose from the requirement to improve 
efficiency and customer service by standardizing DoD processes, standardizing 
and sharing cross-functional data, and using current technology. Specifically 
SPS was expected to: 

• facilitate user productivity by eliminating duplicate data entry, labor- 
intensive processes, and duplicate information; 

• provide more accurate fund citation information; 

• provide for paperless contracting; 


3 DoD components were the Army, Navy, Air Force, Defense Logistics Agency, and other 
Defense agencies. 
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• improve management reporting and workload management; 

• increase efficiency of contract actions by automation and provide an 
automated environment where electronic commerce 4 is the standard 
interface with industry; 

• reduce problem disbursements in contracting; 

• improve visibility and access to contract and contractor performance 
histories through on-line data; and 

• provide on-line audit trails of contract data. 


Guidance on Performance Measures 

Performance measures are required for all investments in information 
technology. Effective performance measures help ensure that information 
technology delivers the intended benefits. 

Clinger-Cohen Act. The Clinger-Cohen Act of 1996 requires DoD to 
evaluate the results of investments in information technology. The Act requires 
DoD to prescribe performance measurements for all information technology 
acquired. Performance measurements gauge how well the information 
technology supports specified mission requirements and should be designed to 
ensure that investments in information technology provide measurable 
improvements in mission performance. In addition, a system of milestones 
should measure progress on the capability of information technology to meet 
specified requirements. 

Executive Order 13011. Executive Order 13011, “Federal Information 
Technology,” July 16, 1996, requires performance measures to be aligned with 
the DoD performance plans under the Government Performance and Results Act 
of 1993. The 2001 DoD Performance Plan established a performance measure 
for reforming information technology management. This performance measure 
links investments in information technology to mission goals. 

Office of Management and Budget Circular No. A-130. The Office of 
Management and Budget Circular No. A-130, “Management of Federal 
Information Resources,” February 8, 1996, requires management oversight 
of information systems to ensure that each information system meets agency 
mission requirements, meets user requirements, and delivers intended 
benefits. The management oversight should provide periodic reviews of 
information systems to determine whether the systems fulfill mission 
requirements. The oversight includes systematic measures of mission 
performance and post-implementation reviews to validate mission 
performance. The post-implementation reviews should assess whether the 
information technology meets the original objectives and achieves the projected 


Electronic Commerce is the interchange and processing of information using electronic 
techniques for accomplishing transactions. 
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benefits. Post-implementation reviews provide management a baseline for 
deciding whether to continue without adjustment, to modify the system to 
improve performance, or, if necessary, to consider alternatives to the 
implemented system. 

Operation of the Defense Acquisition System. DoD Instruction 5000.2, 
“Operation of the Defense Acquisition System,” October 23, 2000, establishes a 
framework for translating mission needs into stable, well-managed acquisition 
programs. The framework includes a series of milestones that must be met for 
the acquisition program to advance to the next phase of the acquisition process. 
The Milestone Decision Authority is the individual authorized to approve an 
acquisition program’s entry into the next phase. However, for information 
systems, the Milestone Decision Authority cannot approve an acquisition 
program’s entry into the next phase until certain actions are completed. One 
required action is a written confirmation by the Chief Information Officer of the 
DoD Component responsible for the information system. The written 
confirmation must indicate that: 

• the information system is being developed in accordance with the 
Clinger-Cohen Act, and 

• performance measures, which are mission-related and outcome- 
based, have been developed and linked to strategic goals. 

In addition, for information systems such as SPS, which are being 
implemented in phased, successive increments, the confirmation must state 
that each increment: 

• meets part of the mission need, independent of future 
increments, and 

• delivers a measurable benefit, independent of future increments. 


SPS User Satisfaction 

The PMO has taken steps to better identify and meet user needs, and 
respondents stated that SPS has the potential of being a very effective and useful 
tool. However, although 85.9 percent of SPS users stated that SPS was 
available always or most of the time, the audit generally substantiated the 
concerns expressed by the Chairman, House of Representatives Committee on 
Budget. According to the survey respondents who stated that they used SPS 
version 4.1, more needed to be done to improve user satisfaction in the areas of 
SPS functionality, implementation, and operation. The following discussion 
provides details on the survey questions, projected survey results for users who 
used SPS version 4.1, and additional information provided by survey 


7 



respondents in essay questions. 5 Also included in the discussion is the 
information obtained based on Joint Interoperability Test Command (JITC) 
Operational Assessments of SPS performed from March through July 2000. 

Prior Audit Identified Functionality Issues. As a result of Inspector General, 
DoD, Report No. 99-166, the SPS Program Manager acknowledged the 
concerns of SPS users regarding the adequacy of critical functional requirements 
and took immediate steps to address user concerns. Specifically, in May 1998 
the SPS PMO established a Joint Requirements Board to reevaluate deficiencies 
identified by SPS users and changes needed in SPS to meet user needs. 
According to the DDP and PMO officials, the Joint Requirements Board was at 
that time addressing 36 additional enhancements identified by users. The Joint 
Requirements Board met monthly to address user requirements and concerns. 

Improvement to SPS Functionality. According to the survey of users, the 
projected survey results, shown in Table 1, indicated that overall functionality 
continued to be an area that needed improvement. Although about 62 percent 
of users indicated SPS had all or most of the functionality needed, about 
38 percent of the users indicated that SPS had only some or none of the 
functionality needed. 


Table 1. “Does SPS have the functionality 


that is needed for you to do your job?” 


Answer Choice 

Percent 

1 All of the functions I need 

12.6 

2 Most of the functions I need 

49.0 

3 & 4 Some or none of the functions needed 

38.4 

Total 

100.0 


The survey results also indicated whether SPS had the functionality necessary 
for respondents to do their jobs. The results varied by user group based on raw 
data. The respondents were assigned to user groups based on DoD Component, 
job title, version 4.1 maintenance release used, and length of use. Respondents 
most satisfied with functionality were system administrators. 

In addition, respondents reported that they were satisfied with some areas of 
SPS functionality. In the essay portion of the survey, respondents indicated the 
best features were the document management capability and that SPS was a 
Windows-based system. In addition, based on raw data, users were satisfied 
with edit checks. 

In the essay portion, respondents reported that they were not satisfied with some 
areas of SPS functionality and indicated some capabilities that needed 


5 The additional information from the essay questions provided in this report reflect comments 
received from multiple users. 
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improvement. Specifically, respondents identified needed improvements for 
fund citations, report generation, contract clause selection, electronic 
transmission, historical data access, indefinite-delivery contracts, contract 
modifications, and saving and printing documents. 

Fund Citations. SPS was expected to eliminate labor-intensive 
processes and provide more accurate fund citation information. However, SPS 
did not have the ability to search, edit, or globally change fund citations. 
Therefore, respondents stated that they used a manual workaround to select fund 
citations; however, the workaround was time-consuming, as users had to 
manually scroll down through numerous fund citations for each contract line 
item. Respondents suggested that functionality could be improved with a query, 
edit, and global change capability. JITC Operational Assessments also 
identified a need for a search feature for fund citations. 

Report Generation. One of the expected benefits of SPS was to 
improve management reporting capability and workload management. 
Respondents stated that the SPS reporting capability was inadequate for accurate 
management and workload data, including lead-time statistics. In addition, 
respondents reported that their legacy systems were used as a workaround to 
generate reports. Respondents suggested the functionality of SPS be improved 
so that reports could be tailored for management needs. JITC Operational 
Assessments also reported a need for increased report generation functionality. 

Contract Clause Selection. When asked about the adequacy of the 
contract clause selection, raw data indicated respondents who used that 
functionality were about evenly split between those who said the clause selection 
was inadequate and those who said that the clause selection was adequate or 
better. Numerous respondents reported that SPS did not automatically select the 
appropriate contract clauses. Respondents also reported that SPS generated 
duplicate clauses. In addition, clauses were not put in the correct order, which 
forced users into a manual workaround of cutting and pasting the clauses into 
the correct order. Respondents suggested that this labor-intensive process could 
be simplified if SPS automatically listed clauses in the proper order. In 
addition, respondents suggested a clause template. JITC Operational 
Assessments also identified a need for improved clause selection functionality. 

Electronic Transmission. Respondents reported a need for 
improved functionality for electronic transmission. Of the respondents who had 
transmitted files to contractors, raw data indicated that more than one-half stated 
the electronic transmission functionality was not adequate. One of the 
objectives of SPS was to facilitate user productivity through the elimination of 
duplicate information and to provide an automated environment for electronic 
commerce. However, SPS did not meet these objectives and required manual 
workarounds. For example, to send a document to a contractor, respondents 
indicated that they had to save the SPS generated document into Microsoft (MS) 
Word and then send the contractor an e-mail with the MS Word file attached. 
Because the MS Word file was read only, the contractor could not use the MS 
Word document to submit a proposal. In addition, this process did not work for 
modifications. Therefore, contracting officials either processed modifications in 
SPS and sent a paper copy to the contractor or processed the modifications 
without SPS and sent an electronic copy to the contractor. Further, drawings 
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and specifications that were larger than 8.5 by 11 inches could not be input into 
SPS for electronic transmission. JITC Operational Assessments also identified 
a need for an SPS capability to import large documents and electronically 
transmit documents. 

Historical Data Access. Respondents reported a need for improved 
historical data functionality. SPS was supposed to improve visibility and access 
to contract and contractor performance information through on-line contractor 
past performance histories. However, respondents reported that the SPS 
historical data capability was inadequate because historical data could not easily 
be viewed and searched. Respondents, including respondents who had used SPS 
for more than 12 months, reported that they used legacy systems as 
workarounds to obtain historical data. Respondents suggested that the 
functionality of SPS could be improved with an independent view and search 
capability. In addition, respondents suggested that the search capability include 
more variables. JITC Operational Assessments also reported a need for an 
enhanced search capability for contract and contractor performance information. 

Indefinite-Delivery Contracts. Respondents reported a need for 
improved functionality for indefinite-delivery contracts, 6 including construction, 
architect, and engineer contracts. Respondents reported that SPS lacked the 
necessary functionality for contract award, delivery orders, task orders, and 
tracking. Therefore, respondents reported using time-consuming workarounds, 
including manual processes, to compensate for the lack of SPS functionality. 

Respondents suggested that SPS be enhanced to handle indefinite-delivery 
contracts efficiently. JITC Operational Assessments also reported a need for an 
enhanced functionality for indefinite-delivery contracts. 

Contract Modifications. One of the expected benefits of SPS was to 
increase efficiency of contract actions through automation. However, 
respondents stated that processing contract modifications in SPS was time- 
consuming, required workarounds, and were difficult to track. In addition, 

JITC Operational Assessments indicated that contract modifications, which 
previously required hours to complete, required days to complete with SPS. 
Therefore, some respondents reported that they did not use SPS for contract 
modifications. However, not using SPS for contract modifications defeated one 
of its expected benefits—development of on-line audit trails of the contract 
process and contract data. In addition, because workarounds were being used to 
process a contract modification, an on-line audit trail through SPS was 
not available. 

Respondents also reported that SPS caused unnecessary modifications to 
contracts. Respondents stated that, if they found an error in a document that 
had been released in SPS but not issued, the only way to correct the document 
was to issue a modification because the document could not be recalled in SPS. 


6 Federal Acquisition Regulation subpart 16.5, “Indefinite-Delivery Contracts,” provides that 
indefinite-delivery contracts are used to acquire supplies and/or services when the exact time and 
exact quantities of future deliverables are not known at the time of contract award. There are 
three types of indefinite-delivery contracts: definite-quantity contracts, requirement contracts, 
and indefinite-quantity contracts. 
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Respondents suggested that the functionality of SPS should be improved to 
include the ability to reverse the release of a document before the contract is 
issued. JITC Operational Assessments also identified a need for a capability to 
unrelease documents when changes are required before distribution. 

Saving and Printing Documents. Respondents indicated a need for 
improved save functionality. One of the expected benefits of SPS was increased 
efficiency through the elimination of duplicate data entry. However, 
respondents indicated that some changes, which could only be made while using 
the document view feature, could not be saved. Therefore, SPS required a 
manual workaround that involved duplicate data entry of changes that could not 
be saved. In addition, respondents indicated that that they could not save in all 
SPS screens without having to exit the screen. This resulted in either 
time-consuming saves or lost data when SPS locked up. The lost data 
necessitated duplicate data entry. Respondents suggested the functionality of 
SPS be improved so that data could be saved quickly at any screen in SPS. 

JITC Operational Assessments also stated that SPS needed an automatic 
save feature. 

Respondents also indicated a need for an improved print functionality. 
Respondents could delete the page breaks only while using the document view 
feature. However, as stated above, changes made while using the document 
view feature could not be saved. Without deleting the page breaks, SPS printed 
only two contract line items per page, resulting in unnecessarily lengthy 
documents or the use of a manual workaround that could not be saved. 
Respondents suggested that the SPS functionality be improved to print multiple 
contract lines per page. 

SPS Implementation. According to our survey of users, more needed to be 
done to improve user satisfaction in the area of SPS implementation. Users 
identified needed improvements in the transition to SPS and training. 

Transition to SPS. The projected survey results indicated that the 
transition to SPS needed improvement. Table 2 shows the projected survey 
results on transition to SPS indicating that approximately 70 percent of users 
experienced many problems in transitioning to SPS version 4.1. 


Table 2. “How extensive were any problems 

encountered during the transition to SPS?” 

Answer Choice 

Percent 

1 & 2 No or few problems 

30.1 

3 Many problems 

69.9 

Total 

100.0 
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Problems encountered during transition to SPS varied by user group based on 
raw data. Respondents who had the most transition problems were users who 
had used SPS for 24 months or more and buyers. Respondents who had the 
fewest number of transition problems were procurement clerks. 

Respondents stated that transition problems arose because of initial system 
downtime and because training classes were attended too far in advance of the 
transition to SPS. By the time SPS was installed, the respondents had forgotten 
much of the information learned during training. Some respondents stated that 
they received training more than 1 year before they transitioned to SPS. When 
training was received too early, respondents requested refresher courses. 
Respondents suggested that training and transition to SPS coincide. A JITC 
Operation Assessment stated that training received too early hindered initial 
effectiveness in using SPS. 

SPS Training. The projected survey results indicated that 
training was an area that needed improvement. Table 3 shows the projected 
survey results on SPS training. Based on the results, training adequately 
prepared about 35 percent of users to operate and understand SPS and training 
did not adequately prepare about 65 percent of SPS users to operate and 
understand SPS. 


Table 3. “Did all the training received adequately 

prepare you to operate and understand SPS?” 

Answer Choice 

Percent 

1 & 2 Yes, completely or to a large extent 

35.4 

3 To a small extent 

54.9 

4 Not at all 

9.7 

Total 

100.0 


How well the training prepared respondents to operate and understand SPS 
varied by user group based on raw data. Respondents most satisfied with 
training were system administrators and respondents who had used SPS 
for 24 months or more. Respondents least satisfied with training were 
contracting officers. 

In an essay portion of the survey, respondents indicated specific areas of 
training that needed more coverage. Based on responses to the survey, training 
coverage could have been improved in areas related to contract modifications; 
delivery orders; clause selection; post awards; purchase requests, including 
splitting and combining; small purchases; task orders; and report generation. 
Survey respondents also reported that indefinite-delivery, architecture and 
engineering, and construction contracts were not covered. In addition, the 
respondents stated that more hands-on training was needed, including 
troubleshooting and workarounds. 
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SPS Operation. According to the survey of users, more needed to be done to 
improve user satisfaction in the areas of SPS operation. Users identified needed 
improvement in the areas of SPS guidance and problem resolution resources. 
However, users were generally satisfied with the availability of SPS. 

SPS Guidance. The projected survey results indicated that guidance 
was an area that needed improvement. Guidance included user manuals, release 
notes, product information provided by AMS, standard operating procedures, 
and documented workarounds. Table 4 shows the projected survey results on 
SPS guidance. Users were about equally divided on whether guidance was 
helpful. However, about 16 percent of users indicated that no guidance 
was received. 


Table 4. “To what extent has guidance provided by 
AMS or the SPS Program Office been helpful?” 


_ Answer Choice _ Percent 

1 & 2 Very helpful or helpful 40.6 

3 & 4 A little helpful or not helpful 43.3 

5 No guidance received 16.2 

Total 100.0* 


*Total does not equal sum of percents due to rounding 


Satisfaction with the helpfulness of guidance varied by user group based on raw 
data. Respondents most satisfied with guidance were procurement clerks, 
system administrators, Air Force users, other Defense agency users, and 
respondents that had used SPS for 24 months or more. Respondents least 
satisfied with guidance were buyers, managers, and Army users. 

Respondents identified needed improvements in the user manual. They 
indicated that the user manual was too generic to be practical and did not cover 
all the steps necessary to perform a task. Respondents suggested a need for 
guidance with an index and step-by-step instructions for each type of contract. 
Respondents stated that error messages generated by SPS were not 
understandable or explained in any guidance. JITC Operational Assessments 
also stated that the error messages were difficult to understand. 

SPS Problem Resolution. The projected survey results indicated 
that problem resolution was an area that needed improvement. Table 5 shows 
the projected survey results on SPS problem resolution. Results indicate that 
available problem resolution resources were not adequate for approximately 
54 percent of users. 
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Table 5. “Overall, when you have a problem, are 
the available resolution resources adequate?” 


Answer Choice 

Percent 

Completely or to a large extent 

45.3 

To a small extent 

46.3 

Not at all 

8.4 

Total 

100.0 


Satisfaction with the adequacy of help resources available varied by user 
group based on raw data. Respondents most satisfied with the help resources 
were users who had used SPS for 24 months or more, procurement analysts, 
and system administrators. Managers were the least satisfied with the 
help resources. 

Respondents identified needed improvements in problem resolution. 

Specifically, respondents identified needed improvements for remote sites, the 
help desk, and the SPS on-line help feature. 

Remote Sites. Respondents at remote sites with databases and 
offsite system administrators indicated that problem resolution resources were 
not adequate. For these sites, the problem resolution resources had not 
adequately addressed downtime due to connectivity problems. Because DoD 
Components were in the process of moving toward regional administration of 
SPS, which will result in more remote sites, it is important that the issue of 
problem resolution at remote sites be addressed. 

Help Desk. Respondents indicated needed improvements at 
the help desk. The most common complaint was that numerous calls were 
required before help was provided. In addition, respondents indicated that they 
were told to perform the process again, which resulted in duplicate data entry. 

SPS On-Line Help Feature. Respondents indicated that the 
SPS on-line help feature needed improvement. The on-line help feature did not 
adequately resolve problems. 

SPS Availability. The projected survey results indicated that SPS is 
generally available. Table 6 shows the projected survey results on SPS 
availability. The survey results indicated that for about 86 percent of SPS users, 
SPS was available always or most of the time. 
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Table 6. “How often is SPS functional and available for your use?” 

Answer Choice 

Percent 

1 Always 

12.7 

2 Most of the time 

73.2 

3 & 4 Down more than up or never available 

14.1 

Total 

100.0 


Satisfaction with the availability of SPS varied by user group based on raw data. 
Respondents most satisfied with availability of SPS were Air Force users. 

Some respondents stated that the biggest problem with SPS availability was lock 
ups and Dr. Watson 7 errors, which necessitated duplicate data entry. When SPS 
locked up and Dr. Watson errors occurred, data were lost and documents had to 
be recreated and information reentered, resulting in duplicate data entry. In 
addition, when SPS was not available, contract actions were performed outside 
of SPS using Form Flow. 8 Then when SPS was available, the contract actions 
performed outside SPS had to be re-entered in SPS, resulting in duplicate 
data entry. 

Other respondents indicated that the availability problems with SPS were due to 
connectivity to the server. The concern with connectivity was a common 
concern with respondents with an offsite database and/or server. Because DoD 
Components were moving toward regional administration, which will result in 
offsite databases and/or servers, it is important that connectivity from remote 
sites be addressed. 


Deployment and Performance Measures 

According to the survey, more needed to be done to improve user satisfaction 
because the: 

• PMO prematurely deployed SPS, 

• PMO had not developed performance measures to track whether SPS 
met the mission objectives and delivered intended benefits, and 


7 Dr. Watson is a program error debugger that detects and diagnoses program errors and then logs 
the resulting diagnostic information. In the event of a program error. Dr. Watson starts 
automatically. Technical support personnel can then use the information logged by Dr. Watson 
to diagnose problems. 

8 Form Flow is a software package created and marketed by the Jet Form Corporation. The 
program provides blank forms which can be filled in with the necessary information. 
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• DoD Components did not effectively coordinate training and the 

transition to SPS. 

Premature Deployment. Respondents indicated that problems with SPS 
functionality were due to premature deployment. Respondents felt that testing 
was performed by the SPS users after deployment and more testing should have 
been done prior to the deployment of SPS. Although system testing generally 
will not identify all problems with a system, adequate testing will minimize 
problems when the system is deployed. In 1997, JITC conducted an initial 
Operational Test and Evaluation of SPS Increment 1 (SPS version 3.1) and a 
follow-on Operational Test and Evaluation of SPS Increment 2 (SPS 
version 3.5). In 1998, JITC conducted a follow-on Operational Test and 
Evaluation of SPS Increment 3 (SPS versions 4.0 and 4.1). In all three 
assessments, JITC concluded that these increments were not operationally 
effective or operationally suitable, except for contracting offices having little or 
no existing automated procurement support. However, despite the JITC 
assessments, the PMO deployed SPS increments 1 through 3 to sites where SPS 
was not operationally effective or operationally suitable. SPS was deployed 
despite the JITC assessments in order to retire legacy systems before year 2000 
and to address DoD problem disbursements. To prevent further premature 
deployment, the PMO should perform adequate testing of the functionality of 
future versions to ensure that they meet the needs of the intended users. 

Performance Measures. The PMO had not developed performance measures 
to track whether SPS met the mission objectives and delivered intended benefits. 
Although the PMO developed system performance and capability measures, 
those measures were not adequate. Therefore, the PMO needs to develop and 
evaluate additional performance measures or risk developing a procurement 
system that does not meet agency mission requirements, meet user 
requirements, or deliver intended benefits. 

Existing Performance Measures. Although the PMO developed 
system performance and capability measures in the Operation Requirements 
Document, April 6, 1998, the audit clearly indicated that those measures were 
not adequate to determine whether SPS met mission requirements, user 
requirements, and delivered intended benefits. For example, there were 
thresholds for data accuracy, data relevancy, data currency, edit checks, and 
data integrity. However, there were no measures to gauge whether SPS was 
providing measurable improvements in mission performance. In addition, 
although there was a threshold for SPS availability, the threshold excluded 
downtime caused by connectivity problems. Therefore, the threshold did not 
measure the actual availability of the system for users. Further, although the 
PMO indicated that the quarterly Defense Acquisition Executive Summaries 
contained information on performance measures, those measures were not 
adequate as the measures were similar to the performance measures in the 
Operation Requirements Document. In addition, the performance characteristics 
reported in the quarterly Defense Acquisition Executive Summaries did not 
differentiate between different SPS versions and maintenance releases. 

Development of Performance Measures. The PMO had not 
developed performance measures to determine whether SPS is incrementally 
meeting the mission objectives and delivered intended benefits. To adequately 
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manage risk, it is necessary to develop performance measures early in the 
information system life cycle. Effective performance measures reduce risk and 
help ensure that each version of information technology incrementally meets 
mission objectives and delivers intended benefits. Although the SPS mission 
objectives and intended benefits are documented, the PMO did not develop 
performance measures to gauge whether the mission objectives and intended 
benefits were incrementally being achieved. The SPS Mission Need Statement 
documented the mission needs and intended benefits. The SPS Economic 
Analysis quantified the intended benefits and concluded that SPS would be a 
cost-effective solution to the mission needs. 

Mission Need Statement. Although the Mission Need 
Statement documented the mission needs and intended benefits of SPS, the PMO 
did not develop performance measures for all the mission needs and intended 
benefits in the Mission Need Statement. For example, there are no performance 
measures for the following mission needs and intended benefits of SPS: 

• facilitating user productivity through elimination of duplicate 
information; 

• providing an automated environment for electronic commerce, 
including the capability to exchange data within DoD and 
with industry; 

• improved access to contract and contractor performance histories; 

• facilitating end user productivity through elimination of redundant 
databases, data transmission, and duplication of information; 

• increasing efficiency of contract actions through automation; 

• eliminating paper handling tasks and achieving paperless contracting, 

• enhanced customer service; and 

• reduced problem disbursements. 

Economic Analysis. Although the SPS Economic Analysis 
quantified intended operational benefits of $1.4 billion, the PMO did not 
develop performance measures for all of the intended operational benefits. For 
example, there were no performance measures for the following operational 
benefits, which accounted for 57 percent of the intended benefits. 

• A productivity increase due to the graphic user interface 9 was 
estimated to achieve $486 million in intended benefits, which was 
equal to 35 percent of the total intended benefits. 


9 A user interface that displays in graphic or pictorial format rather than in text only. 


17 



• A reduction in paper and file handling costs for paper and files, 

paper and photocopy costs, and floor space for storing files was 

estimated to achieve $305 million in intended benefits, which was 

equal to 22 percent of the total intended benefits. 

Evaluation of Performance Measures. To adequately manage risk, 
it is necessary to periodically evaluate performance measures throughout the 
information system life cycle. The Clinger-Cohen Act of 1996 requires DoD to 
monitor and evaluate information technology programs using performance 
measures to determine whether to continue, modify, or terminate a program. 
Periodic evaluations identify problems in a timely manner and allow DoD to 
take prompt corrective action when necessary. Post-implementation evaluations 
are required to assess whether the information system meets the original 
objectives and achieves the intended benefits. For incrementally developed 
systems like SPS, the post-implementation evaluations allow DoD to determine 
whether SPS is incrementally meeting mission requirements, user requirements, 
and delivering intended benefits that were used to justify the information 
system. At a minimum, performance measures should be reviewed after each 
version or maintenance release is deployed. The post-implementation review 
would allow management to adequately address the risks before deploying a new 
version or maintenance release. 

Risks from Performance Measures. The risks for developing a 
procurement system that does not meet agency mission requirements or user 
requirements, and does not deliver intended benefits are increased without 
periodic evaluations of performance measures. In addition, the risk increases 
the longer it takes to validate mission performance using performance measures. 
In order to minimize risks, the PMO should develop quantifiable performance 
measures and use them to evaluate each new version and maintenance release of 
the SPS to ensure that SPS is incrementally meeting the mission objectives, 
delivering the intended benefits, and providing measurable improvements to 
mission performance. If the performance measures indicate that the SPS is not 
incrementally meeting mission and user requirements, the PMO should 
minimize the risks by determining corrective actions to ensure measurable 
improvements to mission performance. By using performance measures to 
evaluate mission performance and minimize risks, DoD can assure that the final 
version of SPS will meet all mission needs and achieve all the benefits that were 
used to justify SPS. 

Coordination of Training and Transition to SPS. The DoD Components did 
not effectively coordinate training and transition to SPS. Respondents indicated 
that training was received too far in advance of the transition to SPS. It is 
important to carefully coordinate training and the transition to SPS because the 
effectiveness of training is reduced when held too far in advance of using a new 
computer system. The DoD Components and PMO should coordinate the 
training and the transition to SPS and provide refresher courses if training is 
held too early. 
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Results of SPS Use 


The projected survey results indicated that: 

• 60.8 percent of SPS users preferred a procurement system other 
than SPS, 

• 45.8 percent of users stated that workarounds increased, 

• 51.4 percent of SPS users stated that productivity does not exceed 
productivity before SPS version 4.1, and 

• 63.5 percent of SPS users stated that SPS had not substantially 
contributed to the DoD goal of paperless contracting. 

The following discussion shows the survey questions related to the impact of 
user satisfaction, projected survey results for users who used SPS version 4.1, 
and supporting information provided by survey respondents in essay questions. 

Preferred Procurement System. The projected survey results indicated that a 
majority of SPS users preferred a legacy procurement system or other means to 
perform procurement functions other than SPS. Table 7 shows the projected 
survey results on the preferred procurement system. Specifically, while about 
40 percent preferred SPS, about 60 percent preferred a system other than SPS. 


Table 7. “If you had a choice of procurement systems, 
would you choose?” 



Answer Choice 

Percent 

1 

SPS 

39.2 

2 

Legacy system 

31.5 

3 

Other 

29.3 


Total 

100.0 


The procurement system of choice varied by user group based on raw data. 
Respondents most likely to choose SPS as their system of choice were 
procurement clerks and system administrators. 

The projected survey results indicated that about 60 percent of SPS users would 
choose a legacy procurement system or other means to perform procurement 
functions other than SPS. That may have reflected respondent dissatisfaction 
with the functionality and operation of SPS. In essays, respondents stated that 
SPS had less functionality, required more steps, and was slower than their 
legacy system. In addition, all necessary interfaces had not been built, such as 
interfaces with supply and accounting systems. 
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SPS Workarounds. The projected survey results indicated that workarounds in 
SPS increased in comparison to the use of workarounds with the users’ legacy 
systems. Table 8 shows the projected survey results on SPS workarounds. 
Specifically, about 45 percent of users indicated an increase in the number of 
workarounds since implementing SPS as compared to 31 percent who stated the 
number of workarounds with SPS were about the same or less than those used 
with a legacy system. 


Table 8. “Compared to your legacy system (previous automated system), 
how many workarounds are you using in order to 
make SPS function appropriately for your site?” 



Answer Choice 

Percent 

1 

Did not use a legacy system 

22.9 

2 & 3 

A lot more or more 

45.8 

4 

About the same 

16.3 

5 & 6 

Less or a lot less 

15.0 


Total 

100.0 


The number of workarounds varied by user group based on raw data. 
Respondents who reported the least workarounds were procurement clerks. 
Respondents who reported the most workarounds were managers and Air 
Force users. 

Respondents indicated that the lack of functionality resulted in time-consuming 
workarounds. First, respondents waited for the workarounds to be developed. 
Then respondents performed the workarounds, which were often time- 
consuming. Respondents reported that workarounds were required for all 
phases of the procurement process including fund citations, report generation, 
contract clause selection, electronic transmission, historical data access, 
indefinite-delivery contracts, contract modifications, and saving and printing 
documents. JITC Operational Assessments also reported concerns with the 
number of workarounds and the time required to perform the workarounds. The 
increase in workarounds was contrary to the DoD Functional Area Reform Goal 
of reforming the information technology management processes to increase 
efficiency and mission contribution. 

The workarounds included manual changes to SPS generated documents. For 
example, when SPS did not correctly number a modification, users manually 
changed the modification number. In a paper-based procurement process, 
manual changes to computer-generated documents may have been acceptable, 
but as DoD moves to paperless contracting, manual changes cannot not be used 
to compensate for system deficiencies, without jeopardizing the intended 
benefits of electronic transmission. 


20 




Productivity. The projected survey results show that the level of user 
productivity does not exceed productivity levels achieved before the introduction 
of the SPS, 4.1 series. Table 9 shows that about 13 percent of users stated that 
productivity increased, 36 percent stated productivity was about the same, and 
51 percent of users reported that productivity had not increased since SPS 
version 4.1 was implemented. 


Table 9. “Since the deployment of the SPS 4.1 series:” 


Answer Choice 

Percent 

1 &2 

Productivity increased immediately or 
productivity initially decreased but has 
since increased and now exceeds 



productivity before SPS, 4.1 series 

12.7 

3 

About the same 

35.9 

4 & 5 

Productivity decreased or productivity 
initially decreased but has since increased, 
however, productivity still does not exceed 



productivity before SPS, 4.1 series 

51.4 


Total 

100.0 


The impact of SPS on productivity varied by user group based on raw data. 
Respondents who were most satisfied with the level of productivity were Navy 
users. Respondents who were least satisfied were Air Force users. 

The projected survey results indicated that a majority of SPS users stated that 
productivity had not increased which was contrary to the DoD Functional Area 
Reform Goal of reforming the information technology management processes to 
increase efficiency and mission contribution. The decrease in productivity 
reflected the lack of SPS functionality and problems in implementation and 
operation of SPS. For example, one respondent, who had used SPS for more 
than 1 year, stated that SPS takes 25 to 30 percent longer. Another respondent, 
who had used SPS for more than 6 months, stated that an action that would take 
15 to 30 minutes in the legacy system took 2 to 2.5 hours in SPS. 

Paperless Contracting. According to the survey, SPS had not substantially 
contributed to the DoD goal of paperless contracting. Table 10 shows the 
projected survey results on paperless contracting. Specifically, about 37 percent 
of users reported that SPS allowed them to move completely, or to some extent 
toward, paperless contracting while about 63 percent of users reported that SPS 
allowed them to move a small extent or not at all toward paperless contracting. 
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Table 10. “Has SPS allowed you to move 
toward paperless contracting?” 

Answer Choice 

Percent 

1 & 2 Completely or to a some extent 

36.5 

3 To a small extent 

35.8 

4 Not at all 

27.7 

Total 

100.0 


The impact of SPS on paperless contracting varied by user group based on raw 
data. Respondents who reported the greatest movement to paperless contracting 
were users who had used SPS for more than 24 months and system 
administrators. Respondents who reported the least movement to paperless 
contracting were buyers, procurement clerks, and other Defense agency users. 

The projected survey results indicated that SPS allowed only about 37 percent of 
SPS users to move completely or to some extent toward paperless contracting. 
However, the DoD Functional Area Reform Goal is to decrease paper 
transactions by 50 percent. Respondents stated that SPS did not contribute 
substantially toward paperless contracting because of the lack of functionality. 

In addition, to meet the DoD goal of being 90 percent paperless by January 1, 
2001, some users stopped using SPS for contract modifications. Users stated 
that they did not use SPS for contract modification because modifications 
generated in SPS could not be electronically transmitted. 

The projected survey results differed substantially from the published DoD 
reports on paperless contracting, which indicated that more than 67 percent of 
contracting actions were paperless. The difference between the survey results 
and the DoD reports was attributable to how paperless contracting actions were 
counted for DoD reports. For example, the Army web-based paperless 
contracting metrics guidance provided that a signed paper copy is counted as 
paperless if the paper copy was created electronically with an automated 
contracting system. The Navy web-based paper-free acquisition metrics guide 
provided that a signed paper copy is counted as 90 percent paperless. The 
Air Force web-based instruction provided that a signed paper copy is counted as 
paperless. The Air Force also counted transactions as paperless when 
documents were both transmitted electronically and via a paper copy. For 
example, if a vendor requested a paper copy of a solicitation, the transaction 
was counted as paperless if the solicitation was also made available via other 
non-hard copy means. For DoD reporting purposes, some paper transactions 
are counted as paperless. The survey respondents stated, however, that they 
actually dealt with the paper copies, and, therefore, indicated that SPS had not 
substantially contributed to paperless contracting. 

Some respondents indicated an increased use of paper since SPS was 
implemented. This was because SPS printed only two contract line items per 
page, making documents longer than documents created in legacy systems. 
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To validate that SPS is achieving the goals of paperless contracting, the DDP 
needs to establish a performance measure to measure SPS progress in meeting 
that goal. 


Conclusion 

Implementation of any new system as far-reaching and complex as SPS can 
expect initial difficulties in implementation, user resistance, and user 
dissatisfaction. Since our last report, the PMO has worked hard to coordinate 
with the DoD Components to address user concerns. However, except for 
positive perceptions on SPS reliability, the user community remained 
fundamentally dissatisfied, and more needed to be done to promote greater 
acceptance and user confidence in the areas of SPS functionality, 
implementation, and operation. The PMO tested and accepted SPS, and DoD 
Components determined whether to deploy SPS. However, respondents to the 
survey stated that testing was inadequate to determine whether SPS was 
operationally effective or operationally suitable for their sites. In addition, the 
lack of effective performance measures affected management ability at the DoD 
Component level to make deployment decisions based on whether SPS would 
meet the intended mission objectives and deliver the intended benefits. 
Appropriate performance measures would enable DoD Components to make 
more effective deployment decisions. Using performance measures would help 
assess whether each new version of SPS was incrementally improving mission 
performance so that, by the time the final version of SPS is deployed, DoD 
either will be assured that all SPS objectives are met or can consider 
supplementary or alternative tools for the procurement workforce. 

At the outset, the DDP and the PMO accepted the initial functionality shortfalls 
in an effort to get SPS deployed, particularly to those sites that still relied on 
manual processing of procurement requirements. Although their motives to 
assist procurement officials in performing their jobs more efficiently may be 
laudable, the survey indicated that SPS version 4.1 still has not closed the 
functionality gap. The degree to which version 4.2 will close the gap and 
generate greater user satisfaction remains to be seen. The PMO needs to 
monitor testing, implementation, and user satisfaction to ensure that version 4.2 
and any subsequent versions deliver the intended benefits. 

The acquisition approach called modular or spiral development poses certain 
advantages in terms of putting advanced technology into the hands of users 
quickly. The SPS program’s experience has shown, however, that user 
acceptance of this approach may not be easy to attain. 
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Recommendations, Management Comments, and 
Audit Response 

A.l. We recommend that the Director, Defense Procurement, direct the 
Standard Procurement System Program Manager to: 

a. Perform adequate testing to ensure that each future version 
has the functionality required to meet the needs of intended users prior to 
the release of each new version of the Standard Procurement System. 

DDP Comments. The DDP concurred with the recommendation. The DDP 
stated that a formal, multi-step testing process is an integral part of the SPS 
acquisition strategy and has been in place since contract award to ensure that 
SPS meets contractual requirements and users needs. Testing has included 
contractor testing prior to release to the Government, Government acceptance 
testing to verify that the software satisfies contract requirements, independent 
operational testing, and user field testing to determine whether the SPS satisfied 
user needs. In addition, user recommended enhancements are referred to a 
structured requirements process for consideration. 

DCMA Comments. DCMA concurred with the intent of the recommendation. 
Like the DDP, DCMA stated that testing is a critical part of PMO and DoD 
Component efforts to ensure the adequacy of each SPS version. The testing 
process begins with the definition of contract requirements by the Joint 
Requirements Board and includes testing during development, acceptance testing 
based on contract requirements, and testing to confirm that SPS operates in the 
representative operational infrastructure. Representatives of the DoD 
Components have been involved in all phases of the testing. Because of the 
existing testing process, DCMA stated that no further action was required. 

Audit Response. Although the DDP and DCMA concurred with the 
recommendation, we do not consider the comments to be fully responsive. We 
agree that testing procedures already exist. However, audit results indicate that 
existing test procedures did not ensure that intended users have the functionality 
needed to perform their jobs. About 38 percent of survey respondents stated 
that SPS version 4.1 had only some or none of the functionality needed. In 
addition, survey respondents believed that testing, which should have been done 
before deployment, had to be performed by SPS users after deployment. 
Furthermore, JITC conducted three operational assessments of SPS, including 
one since the Joint Requirements Board was established to better address user 
needs. In all three assessments, JITC concluded that SPS was not operationally 
effective or operationally suitable, except for sites having little or no existing 
automated procurement systems. Despite the JITC assessments, the PMO 
accepted, and DoD Components deployed SPS to sites where SPS did not have 
the lunctionality for the sites’ missions. 

Further evidence of the shortcomings in present testing procedures was the use 
of numerous and time-consuming workarounds. According to a JITC 
operational assessment, users reported that although new releases of SPS 
corrected problems that required workarounds, the new releases contained new 
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problems that required new workarounds. Users also expressed concern with 
the number, complexity, and inefficiency of the workarounds. Although system 
testing generally will not identify all system problems, adequate or more 
appropriate testing should minimize the need for workarounds once a release 
is deployed. 

Despite existing test procedures, more needs to be done to ensure that future 
versions of SPS are operationally effective and contain the functionality required 
to reasonably meet user needs. 

We request that the DDP provide additional comments to the final report that 
specifically address how testing procedures will be improved to ensure that each 
future version of SPS prior to release has the functionality required to meet 
intended user needs. 

Additional Recommendation. As a result of management comments, we added 
Recommendation A.l.b.4. to clarify the required performance measures. 

b. Develop quantifiable performance measures that gauge 
whether the Standard Procurement System: 

1. Meets the mission objectives. 

2. Increases productivity of contracting personnel. 

3. Achieves the goals of paperless contracting 

as envisioned. 

4. Delivers intended benefits. 

DDP Comments. The DDP partially concurred with the recommendations 
A. 1 .b. 1. through A. 1 .b.3. The DDP stated that SPS was a commercial product 
that is being modularly enhanced to satisfy DoD requirements. As such, the 
SPS acquisition strategy explicitly accepted commercial product performance for 
the initial software release, and subsequent requirements identify the functions 
DoD wants SPS to perform without specifying how SPS must perform those 
functions. In addition, the DDP identified impediments to developing 
performance measures at this time. For example, the SPS Operational 
Requirements Document established requirements to satisfy mission needs; 
however, a productivity performance measure was not practicable because 
contracting activities have different supporting infrastructures that affect 
software performance. Thus the lack of a common baseline did not permit 
establishing meaningful, DoD-wide, productivity measures for SPS. 

The DDP also stated that the real contribution of SPS is its ability to exchange 
information with financial, logistics, and requirements generation systems, and 
suggested that performance measures could be developed once these systems 
come on line. Further, the DDP suggested that performance measures for 
paperless contracting be considered once requirements are better defined for the 
“end-to-end process.” 
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DCMA Comments. DCMA concurred with the intent of the recommendations 
A. l.b. 1. through A. l.b.3. However, DCMA indicated that because 
performance measures already exist and are tested, no further action is required. 
DCMA stated that SPS is a commercial product and performance measures were 
already defined in the Operational Requirements Document. DCMA also 
commented that SPS was not established to achieve productivity enhancements 
for individual DoD Components, but for DoD as a whole and that performance 
measures for specific versions of SPS were not included in the contract. DCMA 
also stated that although SPS version 4.1 has the paperless contracting 
capability, full implementation of paperless contracting is dependent on the 
exchange of data between procurement, logistics, and finance systems, which 
are not yet complete. DCMA stated that no further action was required on the 
recommendations. 

Audit Response. The DDP and DCMA comments were nonresponsive. We 
agree that the Operational Requirements Document established performance 
measures for some operational capabilities. Further, by purchasing a 
commercial product, we may not have the ability to specify how proprietary 
software will perform. However, as reported in Inspector General, DoD, 

Report No. 99-166, “Initial Implementation of the Standard Procurement 
System,” May 26, 1999, the DDP acknowledged that the initial SPS product 
would only accomplish 45 percent of DoD procurement functions, with 
55 percent being accomplished through modifications to the commercial 
product. As such, it is important to ensure that the other 55 percent meets 
user needs. 

Although the Operational Requirements Document included performance 
measures for data accuracy, data relevancy, data currency, edit checks, and data 
integrity, these performance measures did not determine whether SPS meets 
mission needs. As such, existing performance measures have not effectively 
assessed the adequacy of SPS performance within DoD. The SPS Mission Need 
Statement identified goals such as increasing contractor personnel productivity; 
eliminating redundant databases; data transmission, and duplicate information; 
achieving paperless contracting; reducing problem disbursements; facilitating 
electronic commerce; making historical data accessible; and enhancing customer 
service. For example, according to audit results, about 51 percent of survey 
respondents indicated that productivity did not exceed productivity before SPS. 
Survey respondents also stated that SPS has not enhanced customer service. In 
addition to the survey results, a JITC Operational Assessment also reported a 
decrease in customer satisfaction. 

The Mission Need Statement added paperless contracting in 1998, and the SPS 
Economic Analysis identified about $305 million of intended benefits from 
reduced costs for paper and filing, photocopy, and storing paper files. 

According to the survey, SPS paperless contracting capabilities needed 
improvement. For example, survey respondents indicated that to transmit a 
contract electronically, a document must be saved in MS Word, then attached to 
an e-mail rather than being transferred by SPS directly. This requires the 
performance of duplicate steps and creates duplicate files. Also, contracting 
officials either had to process modifications in SPS and send a paper copy to the 
contractor or process modifications without SPS and send an electronic copy to 
the contractor. Further, large drawings and specifications could not be input 
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into SPS and electronically transmitted. As such, a performance measure 
should be initiated to better identify how SPS is achieving the expected cost 
savings and meeting user needs. 

The Clinger-Cohen Act requires that performance measures be established for 
any system to gauge how well a system is meeting mission requirements. Thus, 
to meet Clinger-Cohen requirements as well as to fully assess whether SPS is 
achieving the intended benefits and meeting the mission objectives, the PMO 
needs to develop performance measures for the factors that were used to 
justify SPS. 

We request that the DDP provide additional comments in response to the final 
report that specifically address how it will gauge whether SPS is meeting 
mission needs, increasing productivity, achieving paperless contracting goals, 
and achieving the intended benefits that were used to justify SPS. In addition, 
the DDP should provide comments on Recommendation A.l.b.4. that was 
added to the final report to clarify that the SPS performance measure need to 
determine whether SPS is delivering intended benefits. 

c. Evaluate each new version of the Standard Procurement 
System against performance measures before deploying a new version or 
maintenance release. 

d. Determine corrective actions, when the performance measures 
indicate the Standard Procurement System is not meeting mission 
requirements, user requirements, and delivering intended benefits. 

DDP Comments. The DDP concurred with Recommendations A.l.c. and 
A.l.d. The DDP stated that since contract award, a formal testing process 
has been in place to assure SPS meets the contractual requirements and users 
needs. Testing included contractor testing to assure that SPS is ready for 
release to the Government, Government acceptance testing to verify that the 
software satisfies contract requirements, and independent operational and 
user field testing to determine whether the SPS satisfied user needs. In 
addition, potential enhancements suggested by the user during field testing are 
referred to a structured requirements process that assures user recommendations 
are considered. 

DCMA Comments. The DCMA concurred with the intent of 
Recommendations A.l.c. and A.l.d. However, DCMA stated that if SPS does 
not meet mission requirements nor deliver intended benefits, the Milestone 
Decision Authority is responsible for directing program changes. As such, the 
DCMA stated that no further action was required. 

Audit Response. Although the DDP and DCMA concurred with the intent of 
the recommendations, the comments were not fully responsive. The audit 
results clearly indicate that significant progress still needs to be made for SPS to 
meet the mission needs and deliver measurable benefits. The Milestone 
Decision Authority cannot make an informed decision without adequate 
information. Therefore, to provide the Milestone Decision Authority with a 
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baseline for deciding whether to continue SPS without adjustment or to make 
significant changes to SPS in order to improve performance, the DDP and 
PMO must: 

• develop appropriate performance measures (Recommendation 

A.l.b.), 

• evaluate each new version of SPS against performance measures to 
determine whether SPS is incrementally meeting the mission needs 
and delivering measurable benefits, independent of future versions 
(Recommendation A. 1. c.), and 

• determine corrective actions, if the evaluation of the performance 
measures indicates that SPS is not meeting mission needs and 
delivering measurable benefits (Recommendation A.l.d.). 

Once the above actions are completed, the DDP and PMO can determine 
whether corrective actions should be taken, and what Milestone Decision 
Authority approvals are needed. If SPS cannot fully meet user needs, 
supplementary or alternative tools may be necessary. 

The Milestone Decision Authority is scheduled to decide on the next SPS 
milestone in April 2002. According to DoD Instruction 5000.2, “Operation of 
the Defense Acquisition System,” October 23, 2000, the DCMA Chief 
Information Officer must provide a written confirmation to the Milestone 
Decision Authority that SPS is being developed in accordance with the Clinger- 
Cohen Act, including a description of how each increment of SPS meets mission 
needs and delivers a measurable benefit, independent of future versions. Thus, 
to meet Clinger-Cohen requirements and to attest to the Milestone Decision 
Authority that SPS is incrementally meeting mission needs and delivering 
measurable benefits, the DDP needs to complete actions under 
Recommendations A.l.b., A.l.c., and A.l.d. 

We request that the DDP provide additional comments in response to the final 
report that specifically address how it will evaluate each new version of SPS 
against performance measures to determine whether SPS is incrementally 
meeting mission needs and delivering measurable benefits, and indicating 
whether DDP will seek supplementary or alternative tools if necessary to meet 
mission objectives. 

A.2. We recommend that the Director, Defense Procurement, direct the 
DoD Components to: 

a. Coordinate, with the Program Manager, training and the 
transition to the Standard Procurement System Program so that the 
Standard Procurement System is available for use when personnel receive 
training classes. 

b. Provide refresher courses for individuals receiving training 
too far in advance of the transition to the Standard Procurement 
System Program. 


28 



DDP Comments. The DDP partially concurred. The DDP stated that the DoD 
Components are responsible for ensuring that training at their sites is adequate 
and timely. The DDP further stated that the PMO confers weekly with the DoD 
Components to ensure that training courses are scheduled as close to site 
installations as possible. However, the DDP agreed to remind the DoD 
Components to determine whether any employees require any additional training 
and to coordinate the training requirements and schedules with the PMO. 

Audit Response. The DDP comments were responsive. 
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B. Use of the Standard Procurement 
System 

According to our survey of SPS users, we projected that about 
26.5 percent of individuals identified by DoD Components as licensed to 
use SPS version 4.1 have not used it. Respondents indicated that they 
have not used SPS version 4.1 because the DoD Components: 

• deployed SPS to sites even though SPS did not have the 
functionality needed for those sites’ missions, and 

• deployed SPS to employees that did not require SPS to 
perform their jobs. 

In addition, the DoD Components identified employees as licensed 
to use SPS even though those employees did not have SPS installed 
on their computers. 

As a result, we estimated the PMO spent up to $2.1 million of the 
$7.9 million in license costs on licenses for users who could not or did 
not use SPS. These funds could have been put to better use elsewhere. 
In addition to the license costs, the PMO and DoD Components spent 
time and resources for unnecessary SPS deployments. 


SPS Usage 

According to our survey of users, we projected that about 26.5 percent of 
individuals identified by DoD Components as licensed to use SPS version 4.1 
did not use it. Respondents indicated that, rather than use SPS version 4.1 to 
perform their job, they used an earlier version of SPS, a legacy system, a 
manual process, MS Office products, or other means. 

Respondents who have not used SPS version 4.1 did not do so because the DoD 
Components deployed SPS to sites even though SPS did not have the 
functionality needed for those sites’ missions and deployed SPS to employees 
that did not require SPS to perform their jobs. In addition, the DoD 
Components identified employees as licensed to use SPS even though those 
employees did not have SPS installed on their computers. 

Incremental Deployment. According to the SPS acquisition strategy, SPS 
would be delivered in 4 increments (versions 3.1, 3.5, 4.0, and 4.1) of 
increasing functionality until a total of 299 procurement functions were 
deployed. Increment 1 included 69 of the 299 functions identified as suitable 
for testing and deployment to DoD sites that had limited or no automated 
procurement capabilities (mostly Navy sites). Increment 2 was to undergo 
operational testing while Increment 1 was being deployed; Increment 2 would be 
back-fitted and deployed to sites that had already received Increment 1, as well 
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as to other sites. The practice would be repeated with each increment. DoD 
began receiving version 4.1 in September 1998. As such, the PMO recognized 
and accepted that shortfalls in functionality would be unavoidable. 

Functionality Needs of Sites. The DoD Components deployed SPS version 4.1 
to sites even though SPS did not have the functionality needed for the missions 
at those sites. For example, respondents indicated that there were no SPS users 
at two of the sample sites, including sites that performed procurement functions 
for construction, and architecture and engineering services. Respondents 
indicated that SPS version 4.1 was not used because it did not have all the 
functionality needed for construction, architecture, and engineering 
contracting—which is their mission. 

Operational Assessment Results. The JITC, which conducted Operational 
Assessments of SPS version 4.1 at 24 sites, also found that SPS version 4.1 was 
deployed to sites even though SPS version 4.1 did not have the functionality 
needed for that site’s mission. 10 JITC found three sites that did not use SPS 
version 4.1 and one site that awarded only a small portion of its contracts using 
SPS version 4.1. 

Sites Using SPS. Three sites did not use SPS version 4.1 because 
SPS did not have the functionality needed for those sites’ missions. JITC 
determined that multiple versions of SPS had been deployed to the Space and 
Naval Warfare Systems Command, San Diego, California. That site, which 
procured major weapons systems, had not used SPS because SPS did not have 
the functionality needed for major weapons systems. SPS will not have the 
functionality needed for major weapons systems until version 5.0, which is 
scheduled for deployment in the second quarter of FY 2002. In addition, JITC 
found that since 1997 multiple versions of SPS were deployed to the 
Superintendent of Shipbuilding, Newport News, and Portsmouth, Virginia, 
although those sites did not use SPS because SPS did not have the needed 
functionality for the sites’ missions. 

Contracts Awarded Using SPS. JITC found that the Naval 
Facilities and Engineering Command, San Diego, California, used SPS 
version 4.1 to award only a small portion of its contracts. Prior to deploying 
SPS in August 1999, the Naval Facilities and Engineering Command at San 
Diego had no automated procurement system. Since the SPS deployment, the 
site awarded only 4 contracts using SPS of the approximate 40 to 50 contracts 
awarded per month. This is because SPS did not have the functionality for large 
and complex contracts, which comprise most of the contracts awarded at the 
site. In addition, 40 percent of the site’s contracts were awarded through 
simplified acquisition procedures, but SPS was not used for the simplified 
acquisitions because there was no site requirement to use SPS. Further, users 
were hesitant to invest much time and effort in using SPS because users believed 
that SPS may ultimately not meet the site’s mission needs. 

SPS Needs of Employees. The DoD Components deployed SPS version 4.1 to 
employees who did not require SPS to perform their jobs. For example, SPS 


10 These sites were not included in our sample of sites. 
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was deployed to employees such as engineers, engineering technicians, and 
attorneys although they did not use SPS. However, SPS was intended for use 
by contracting employees and not by engineers or attorneys. In addition, SPS 
was deployed to contracting employees who did not require SPS to perform 
their jobs. For example, SPS was deployed to employees performing functions 
such as administering construction contracts and procurement management 
reviews who stated that they did not need SPS to perform their jobs. SPS will 
not have the functionality needed for contract administration until version 4.2, 
which is scheduled for deployment in the second quarter of FY 2001. 

Identification of Users. The DoD Components identified employees as 
licensed to use SPS even though those employees did not have SPS installed on 
their computers. For purposes of the survey, we requested from the DoD 
Components and were supplied with a list of personnel who, according to DoD 
Component records, received SPS version 4.1 licenses and were considered 
“SPS users.” However, based on results of the survey, the lists were not 
accurate. We contacted individuals that chose not to remain anonymous when 
responding to the survey and stated that they did not use SPS. Of the 
22 individuals who responded, 7 indicated they did not have SPS installed on 
their computers and did not need it to perform their jobs. 


Resources Expended and Implementation of Future Versions 

We estimate that the PMO spent up to $2.1 million of the $7.9 million in license 
costs on unnecessary licenses for personnel who did not use SPS. In addition to 
the license costs, the PMO and DoD Components spent time and resources for 
unnecessary SPS deployments. Furthermore, by deploying SPS without the 
needed functionality to meet sites’ missions, the DoD Components may have 
made users reluctant to implement future versions of SPS. 

License Costs for Sites. The PMO incurred unnecessary license costs when 
SPS was provided to sites even though SPS did not have the functionality needed 
for the sites’ missions. For example, we determined that there were no users at 
two sites because, according to survey respondents, SPS functionality did not 
meet the sites’ missions. For another 11 of the 120 sites in the survey, all 
respondents stated that they did not use SPS. In addition to our survey, JITC 
determined that 3 of the 24 sites included in its Operational Assessments had no 
SPS users. 

License Costs for Employees. The PMO may have incurred unnecessary 
license costs when SPS was deployed to employees who did not require SPS to 
perform their jobs. We cannot precisely quantify the total amount of 
unnecessary license costs because of the way in which licenses were purchased. 
Licenses were purchased in blocks of 1, 2-10, 11-24, 25-50, 51-100, 101-250, 
251-500, and 501-1000. The license cost was also dependent on the office 
suite used (WordPerfect or MS Office), and the hardware platform. For 
example, if a site had 51 users, MS Office, and a Hewlett-Packard 9000 
platform, the license costs for the site were $24,335 in 2000. If the site had 

100 users, the license costs were also $24,335 in 2000. However, if the site had 

101 users, the license costs for the site were $47,171 in 2000. 
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Estimate of License Costs. We estimate that the PMO spent up to 
$2.1 million, or up to 26.5 percent of the $7.9 million in license costs, on 
unnecessary licenses for users who did not use SPS. We estimated the 
unnecessary license costs based on the projection that about 26.5 percent of 
users, identified by DoD Components as licensed to use SPS version 4.1, did 
not use version 4.1. According to the PMO, some of the $2.1 million is time 
value of money as a result of his purchasing licenses before necessary. Based 
on our review of the license costs, although some of the license costs increased 
in the option years, other costs decreased. 

Other Resources. In addition to expending unnecessary funds on licenses, the 
PMO and DoD Components spent time and resources for unnecessary SPS 
deployments. Deployment costs include time and resources for installation, 
travel, and training. In addition, some DoD Components contracted with AMS 
and other contractors for deployment support. For example, Inspector General, 
DoD, Report No. 99-166, “Initial Implementation of the Standard Procurement 
System,” May 26, 1999, stated the Air Force Contracting Information System 
Program Office estimated that DoD Components may spend $70 million for 
additional contractor support for SPS implementation. 

Implementation of Future Versions. By deploying SPS without the needed 
functionality to meet the sites’ missions, the DoD Components may have made 
users reluctant to implement future versions of SPS. The JITC Operational 
Assessments indicated that based on problems encountered with versions 3.1 and 
3.5, users were reluctant to implement version 4.1. By deploying version 4.1 to 
sites even though SPS did not have the functionality needed for the sites’ 
missions, the DoD Components increased the risk that problems encountered 
with version 4.1 will negatively impact user acceptance of future versions 
of SPS. 

Conclusion 

The audit determined that up to $2.1 million was spent on unnecessary license 
costs. Although the PMO has been addressing user needs through the Joint 
Requirements Board and making improved versions available, it is up to the 
DoD Components to ensure that the needed functionality exists before deploying 
SPS. DoD Components should carefully consider decisions to deploy SPS to 
the remaining activities. Before deploying SPS to a site, the DoD Components 
need to first determine whether SPS has the functionality to meet that site’s 
mission. Once DoD Components determine that SPS has the functionality for a 
site’s mission, the DoD Components then need to determine whether individual 
employees require SPS to perform their jobs. Only after these two 
determinations are made can the DoD Components accurately determine the 
number of required licenses. The DoD Components should: 

• document their decision that SPS has the necessary functionality to 
meet the mission needs of the site, 

• document the number of licenses required, and 

• maintain a record of the licenses and make adjustments if the site’s 
license requirements change. 
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Recommendations, Management Comments, and 
Audit Response 

Deleted, Renumbered, Revised, and Redirected Recommendations. As a 

result of management comments, we revised draft Recommendations B.l.a. and 
B.l.c., deleted draft report Recommendation B.2.b., renumbered 
Recommendation B.2.a. to B.2., and revised and redirected Recommendation 
B.2. to the DDP. In addition, we deleted Recommendation B.3.b., which 
resulted in renumbering Recommendation B.3.a. to B.3. 

B. We recommend that the Director, Defense Procurement: 

1. Require DoD Components, prior to any future deployment of 
either a new version or maintenance release, to: 

a. Provide assurance showing that it has determined that the 
Standard Procurement System has the functionality to meet each requesting 
site’s mission. 


b. Determine the number of employees at each site who 
require the Standard Procurement System to perform their jobs. 

c. Provide assurance that the number of licenses requested 
by site is appropriate. 

d. Correct existing records of Standard Procurement System 
licenses or establish such records, and redistribute licenses when employees 
no longer require the Standard Procurement System to perform their jobs. 

DDP Comments. The DDP partially concurred with draft 
Recommendation B. 1. The DDP stated that Recommendations B.l.a. through 
B.l.c. need to be tailored for individual DoD Component organizational and 
mission-related requirements, and therefore, are handled by DoD Components. 
Further, the DDP contends that to require DoD Components to submit 
documentation to the DDP, as specified in draft Recommendations B.l.a. and 
B.l.c., would add to SPS deployment costs, delay deployment of new or 
improved functionality, and be inconsistent with organizational accountability 
and responsibility concepts. For Recommendation B. 1 .d., the DDP stated that 
SPS licenses are not user specific. Therefore, there is no need to redistribute 
licenses within a site. In addition, the transfer of licenses among sites can be 
accomplished within the license agreement or through negotiations. 

DCMA Comments. DCMA partially concurred with Recommendation B. 1. 
DCMA stated that once the PMO accepts the software, it is up to the DoD 
Components to validate that interfaces are operational, that the software works 
in the component architecture and environment, and that SPS meets their users’ 
needs. Once validation has been performed, it is up to the DoD Components to 
request installation and training from the PMO including specifying the sites, 
identifying intended users, and maintaining license distribution. The PMO 
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purchases licenses based on the number of users at each site and redistribution is 
limited by the contractual terms and existing license agreements. Based upon 
this information, DCMA considered the action to be complete. 

Audit Response. We consider the DDP and DCMA comments to be partially 
responsive. We agree that the DoD Components should be responsible for 
determining the necessary functionality of a version and the number of licenses 
needed. However, based upon audit results, more than 26.5 percent of 
individuals identified by DoD Components as licensed to use SPS version 4.1 
have not used it because it lacks the necessary functionality for their missions or 
because SPS is not needed to perform their jobs. The DoD Components have 
not been accurately assessing SPS needs prior to requesting that the PMO 
purchase licenses. For example: 

• SPS was installed for 6 months at two sites but had no users because 
of functionality shortcomings. 

• SPS was deployed to employees such as engineers, engineering 
technicians, and attorneys although they did not require SPS to 
perform their jobs. 

• SPS was deployed to contract administrators who do not require SPS 
to perform their jobs because SPS will not have the needed 
functionality for contract administration until version 4.2 is deployed. 

In addition to audit results, the JITC Operation Assessments identified 
three sites that did not use SPS because of functionality shortfalls, including 
sites that had SPS installed since 1997. We estimate that $2.1 million has 
been wasted. 

If the deployments were properly planned, users would transition to SPS shortly 
after SPS was installed, not years after SPS was installed. Therefore, DoD 
Components need additional guidance or procedures to determine whether SPS 
has the functionality to meet each site’s mission needs to avoid unnecessary 
purchases of licenses. 

The DCMA stated that DoD Components submit planning information for SPS 
deployments as a part of the yearly cycle, while delivery orders for licenses are 
not issued until 2 to 4 weeks before installation. As such, it should be 
reasonable for DoD Components to provide the DDP or PMO with information 
on licensing needs as part of the yearly requirement and make adjustments prior 
to the delivery order. In view of DDP concerns about the administrative burden 
of potentially voluminous documentation, we revised Recommendations B.l.a. 
and B.l.c. to provide reasonable flexibility. We request that the DDP provide 
additional comments to the final report on Recommendations B.l.a. and B.l.c. 

2. Obtain assurance from the DoD Components, that they have, 
prior to any future deployment of either a new version or maintenance 
release, accurately determined that the Standard Procurement System will 
meet each site’s mission and that deployment to the site is appropriate. 
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DDP Comments. The DDP nonconcurred with Recommendation B.2. The 
DDP stated that DoD Components are responsible for determining whether SPS 
meets their mission needs. The PMO is not in the position to validate a DoD 
Component’s decision that SPS meets a site’s mission needs. 

DCMA Comments. DCMA nonconcurred with Recommendation B.2. DCMA 
stated that the PMO should not be responsible for validating the actual number 
of employees to receive SPS and determining whether each DoD Component 
site should receive new versions or maintenance releases for the SPS. Rather, 
the PMO believes it is the responsibility of each site command to perform these 
functions. The DCMA considered the action to be complete. 

Audit Response. We consider the DDP and DCMA comments to be partially 
responsive. We agree that responsibility for determining whether a version of 
SPS meets each site’s mission should be the responsibility of the DoD 
Components. However, the PMO and DDP need to be proactive and question 
how the sites are validating their license requirements. Therefore, we maintain 
that the DDP, as the DoD organization responsible for SPS, should seek 
credible assurance from the DoD Components that requirements have been 
verified before expending funds to obtain additional licenses. We request that 
the DDP provide additional comments to the final report. 

3. Direct the Program Manager to purchase licenses for all future 
deployments only after the DDP validates that the Standard Procurement 
System meets the mission requirement of the site. 

DDP Comments. The DDP partially concurred and stated that licenses are 
required only in the quantities requested by the DoD Components to support the 
sites designated to receive a particular software release. 

DCMA Comments. DCMA nonconcurred and stated that the PMO should not 
validate that an SPS version will meet a site’s mission needs because this is a 
DoD Component responsibility. DCMA considers the action to be complete. 

Audit Response. The comments provided by the DDP and the DCMA were 
not responsive. Although the DoD Components are responsible for identifying 
quantities of licenses required, the audit clearly indicated that additional actions 
are required because the DoD Components requested that the PMO purchase 
unnecessary licenses. Therefore, the PMO should not purchase new licenses 
until the DDP has assurance that licenses to be purchased are needed. This can 
be accomplished by DDP implementing review procedures for Recommendation 
B.2. and then validating the requirements. The minimal costs for review and 
validation will outweigh future costs from acquiring unneeded licenses. We 
request the DDP provide additional comments to the final report to address how 
the DDP will provide the PMO with more accurate information on licensing 
needs of DoD Components. 
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Appendix A. Audit Process 


Scope 

We developed a web-based survey instrument to assess user satisfaction with 
SPS. We then selected a statistical sample of sites from a list of sites provided 
by the PMO and then did a statistical sample of users at those sites. We 
reviewed JITC Operational Assessments conducted between March and 
July 2000. We met with personnel from the office of the DDP, the DCMA, the 
PMO, and the SPS Component Management Offices. We also attended SPS 
program management reviews. 

Limitation to Scope. We did not review the management control program 
related to the overall audit objective because the audit was in response to a 
congressional request. 

DoD-Wide Corporate Level Government Performance and Results Act 
Coverage. In response to the Government Performance Results Act, the DoD 
annually establishes DoD corporate level goals, subordinate performance goals, 
and performance measures. This report pertains to achievement of the 
following corporate-level goals and subordinate performance goals: 

FY 2001 DoD Corporate Level Goal 2: Prepare now for an uncertain future 
by pursing a focused modernization effort that maintains U.S. qualitative 
superiority in key warfighting capabilities. Transform the force by exploiting 
the Revolution in Military Affairs, and reengineer the Department to achieve a 
21st century infrastructure. (Ol-DoD-2) FY 2001 Subordinate Performance 
Goal 2.3: Streamline the DoD infrastructure by redesigning the Department’s 
support structure and pursuing business practice reforms. (Ol-DoD-2.3) 
Subordinate Performance Goal 2.4: Meet combat forces’ needs smarter and 
faster, with products and services that work better and cost less, by improving 
the efficiency of DoD acquisition processes. (Ol-DoD-2.4) Subordinate 
Performance Goal 2.5: Improve DoD financial and information management. 
(Ol-DoD-2.5). 

DoD Functional Area Reform Goals. Most major DoD functional areas 
have also established performance improvement reform objectives and goals. 
This report pertains to achievement of the following functional area objectives 
and goals. 

• Acquisition Functional Area. Objective: Foster Partnerships. 
Goal: Decrease paper transactions by 50 percent through electronic 
commerce and electronic data interchange. (ACQ-2.3) 

• Information Management Technology. Objective: Reform 
information technology management processes to increase efficiency 
and mission contribution. Goal: Institute fundamental information 
technology management reform efforts. (ITM-3.2) 
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General Accounting Office High-Risk Area. The General Accounting Office 
has identified several high-risk areas in the DoD. This report provides coverage 
of the information Management and Technology high-risk area. 


Methodology 

Use of Computer-Processed Data. We relied on internally computer-processed 
data to collect and analyze responses to a web-based user satisfaction survey. 

The web-based survey was developed and administered in-house. Therefore, 
we considered the data to be reliable. 

Statistical Sampling Methodology. The Quantitative Methods Division, Office 
of the Assistant Inspector General for Auditing, DoD, developed a statistical 
sampling plan to obtain information about the perceptions of and experience 
with the SPS through a survey of its users. Quantitative Methods Division 
analysts designed a multistage stratified sample to provide this information. 

Target Population. The target population for the survey comprises 
DoD users of SPS version 4.1. SPS sites control access to SPS and maintain 
registries of their site’s users. The DoD Components do not maintain central 
registries of users. In the absence of such a central source, we collected lists of 
users from selected sites within DoD. To facilitate this collection and the 
associated analysis, we grouped sites into five strata: Army sites, Navy sites, 

Air Force sites, Defense Logistics Agency sites, and other Defense agencies 
sites. The number of sites per stratum varied, as shown in Table A-l. 


Table A-l. Number of Sites Per Strata 


Stratum 

Total 

Sites 

Sample 

Sites 

Army 

290 

30 

Navy 

182 

28 

Air Force 

19 

All 

Defense Logistics Agency 

9 

All 

Other Defense Agencies 

34 

All 

Total 

534 



Sample Design. The survey required the use of two different 
approaches for sampling users within the overall sampling design. Because the 
Army and Navy had large numbers of sites, we statistically selected 30 Army 
and 28 Navy sites and obtained lists of users from each of the sites. We then 
surveyed all the users for those sites with 20 or fewer reported SPS users, and 
statistically sampled 20 users from those sites with 21 or more users. Because 
there were relatively few sites in the Defense Logistics Agency and other 
Defense agencies, and relatively few operational Air Force sites, we obtained 
lists of users from each site within each of the three strata. We then drew 
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separate simple random samples directly from the combined Air Force list of 
users, the combined Defense Logistics Agency list of users, and the combined 
other Defense agencies list of users. 

We designed the sample plan for statistical selection of the SPS users across 
DoD, and through e-mail requested that the selected users complete the 
web-based survey questionnaire to collect their perceptions of SPS functionality 
and user satisfaction with SPS version 4.1. We used two-stage sample plans for 
both the Army and Navy, and simple random sample plans for Air Force, 
Defense Logistics Agency, and other Defense agencies. 

Army. We used a two-stage sample plan for the Army by 
treating the sites as clusters. In the first stage, we randomly selected a sample 
of 30 sites from the population of 290 Army sites. In the second stage, we 
randomly selected a sample of 278 users from the population of 378 users at 
30 selected sites. We received 205 responses out of 278 users surveyed. We 
used the 205 responses from the 30 selected sites to represent those Army SPS 
users who would have responded to a survey of all SPS users. 

Navy. We also used a two-stage sampling plan for the Navy. 
In the first stage, we randomly selected a sample of 28 sites from the population 
of 182 Navy sites. In the second stage, we randomly selected a sample of 
331 users from the population of 578 users at the selected sites. We received 
217 responses out of 331 users surveyed. We used the 217 responses as the 
overall sample size distributed from the 28 selected sites with users to represent 
those Navy SPS users who would have responded to a survey of all SPS users. 

Air Force. We used a simple random sample plan for the 
Air Force. We randomly selected a sample of 101 users from the population of 
748 users and received 68 responses. We used the responses in this stratum to 
project the stratum’s contribution to the overall DoD responding user 
perceptions of and experiences with SPS. 

Defense Logistics Agency. We used a simple random sample 
plan for the Defense Logistics Agency. We randomly selected a sample of 
102 users from the population of 364 users and received 70 responses. We used 
the responses in this stratum to project the stratum’s contribution to the overall 
DoD responding user perceptions of and experiences with SPS. 

Other Defense Agencies. We used a simple random sample 
plan for other Defense agencies. We randomly selected a sample of 101 users 
from the population of 527 users and received 81 responses. We used the 
responses in the stratum to project that stratum’s contribution to the overall DoD 
responding user perceptions of and experiences with SPS. 

Population Used for Analyses. Out of the 913 sample surveys 
solicited, we received 641 responses, including 439 responses from users of SPS 
version 4.1. These responses represented about 6,385 SPS version 4.1 users 
and comprised the framework within which the analyses in Appendix B are 
computed. The methodology used to construct this estimated number of users 
who would have responded to the survey has been described in detail in 
Appendix C. 
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Use of Technical Assistance. We used technical assistance during the audit. 
The Quantitative Methods Division, Office of the Assistant Inspector General 
for Auditing, DoD, provided a statistical sampling plan and analysis of the data 
gathered via the web-based survey. We met with technical experts at the 
Defense Information Systems Agency; Defense Manpower Data Center; and 
Information Systems Directorate, Office of the Director, Administration and 
Information Management, Office of the Inspector General, DoD, to discuss the 
development, administration, and analysis of a web-based survey. 

Audit Type, Dates and Standards. We performed this program audit from 
January through November 2000, in accordance with auditing standards issued 
by the Comptroller General of the United States, as implemented by the 
Inspector General, DoD. 

Contacts During the Audit. We visited or contacted individuals and 
organizations within DoD. Further details are available on request. 


Prior Coverage 


General Accounting Office 

General Accounting Office Report No. GAO/AIMD 98-40 (OSD Case 
No. 1509), “Financial Management, Seven DoD Initiatives That Affect the 
Contract Payment Process,” January 30, 1998 


Inspector General, DoD 

Inspector General, DoD, Report No. 99-166, “Initial Implementation of the 
Standard Procurement System,” May 26, 1999 

Inspector General, DoD, Report No. 96-219, “Allegations to the Defense 
Hotline Concerning the Standard Procurement System,” September 5, 1996 
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Appendix B. Statistical Projections 


We computed statistical projections of SPS version 4.1 users in percentages for 
30 selected responses to the 11 questions for each DoD group and aggregated 
the results throughout DoD. The individual projections for 30 responses were 
computed by using a 99.65 percent confidence level to have a 95 percent 
effective confidence level for the 30 projections when viewed simultaneously. 
We used the Bonferroni 1 approach to compensate for the reduced level of 
significance for individual tests when conducting multiple tests. 

Nonrespondents at both the survey and the individual item level were treated as 
missing-at-random; that is, as though their answers were in the same 
proportions as those of the respondents. The survey responses for two of the 
sampled Navy sites were commingled. To complete the statistical calculations, 
we divided these responses equally between the two Navy sites. 

Two types of approximations were used nonstatistically in the confidence 
interval calculations. Because complete lists of users were not obtained for 
either Army or Navy, we approximated the numbers of users for these Military 
Departments. Also, because not everyone included on the user lists was a user 
of SPS version 4.1, we approximated the numbers of SPS version 4.1 users in 
the DoD Components (see Appendix C). Both of these approximations 
introduced additional uncertainty to the confidence interval calculations in ways 
that cannot be quantified. Therefore, the presented confidence intervals for the 
estimated response percentages are themselves approximate and understate the 
total uncertainty of the estimates. The estimated percentages should be used for 
decision making only if the presented confidence intervals are well within the 
uncertainty acceptable to the decision maker. The projected results for the 
estimated number of respondents are shown in Table B-l. 


The Bonferroni procedure or correction is used when there are multiple statistical tests. When 
there are multiple independent tests—different survey response categories in this instance—the 
error risks accumulate. 
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Table B-l. Statistical Projections 



Point 

Lower 

Upper 


Estimate 

Bound 

Bound 

Use of SPS: 

Do not use SPS* 

26.5 

6.1 

46.9 

SPS Functionality: 

All functions needed 

12.6 

8.1 

17.1 

Most of the functions needed 

49.0 

40.5 

57.5 

Some/none of the functions needed 

38.4 

31.4 

45.3 

SPS Workarounds: 

More or a lot more 

45.8 

36.3 

55.2 

About the same 

16.3 

11.1 

21.4 

Less or lot less 

15.0 

10.0 

20.1 

Amount Paperless: 

Completely/to some extent 

36.5 

27.1 

45.9 

To a small extent 

35.8 

26.7 

44.8 

Not at all 

27.7 

19.6 

35.8 

Installation Problems: 

No/few problems 

30.1 

19.4 

40.8 

Many problems 

69.9 

59.2 

80.6 

Training Adequacy: 

Completely/to a large extent 

35.4 

25.6 

45.2 

To a small extent 

54.9 

45.8 

64.0 

Not at all 

9.7 

5.0 

14.4 

Helpfulness of Guidance: 

Helpful/very helpful 

40.6 

35.1 

46.0 

Little/not helpful 

43.3 

35.4 

51.1 

No guidance 

16.2 

8.7 

23.7 

System Choice: 

SPS 

39.2 

27.9 

50.6 

Legacy system 

31.5 

23.3 

39.7 

Other system 

29.3 

15.6 

43.0 

Productivity: 

Increased 

12.7 

7.5 

17.9 

About the same 

35.9 

25.6 

46.2 

Decreased 

51.4 

41.5 

61.3 

Adequacy of Help Resources: 

Completely or to a large extent adequate 

45.3 

34.7 

55.8 

To a small extent adequate 

46.3 

37.3 

55.3 

Not adequate 

8.4 

4.5 

12.3 

Functional and Availability: 

Always 

12.7 

6.4 

19.0 

Most of the time 

73.2 

64.1 

82.3 

Down more than up/never available 

14.1 

7.0 

21.1 


*The projections for non-SPS users are based on the estimated number of 
listed users (8,867 DoD-wide) against the projections for other questions 
based on SPS version 4.1 users, as explained in Appendix C. 
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The projected results can be interpreted in statistical terms and illustrated by 
taking any projection from above. For example, to interpret the non-SPS users, 
we are 95 percent confident that within the 30 listed projections, between 
6.1 percent and 46.9 percent of the respondents are non-SPS users. The 
unbiased point estimate of 26.5 percent of non-SPS users is the midpoint of the 
statistically estimated range of stated values. 

The results for non-SPS users show a large confidence interval for DoD-wide 
estimates - 6.1 percent to 46.9 percent. This wide confidence interval 
mathematically reflects the wide differences across the DoD Components, which 
it summarizes. The lower bounds range from 2.5 percent to 72.6 percent. The 
upper bounds range from 28.3 percent to 96.6 percent. In addition, the Navy 
has a wide range of uncertainty - 5.6 percent to 60.9 percent. This also 
contributes to the large DoD-wide confidence interval. By way of contrast, the 
DoD Component intervals for functionality are much closer, with the lower 
bounds from 18.8 percent to 33.1 percent and the upper bounds from 
45.7 percent to 94.8 percent; therefore, the DoD-wide interval is much 
narrower from 31.4 percent to 45.3 percent. 
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Appendix C. Estimating the Number of 

Standard Procurement System 
Version 4.1 Users 


The goal of the SPS user survey was to collect information on user experience 
with, and perceptions of, SPS. We focused our analysis on current users of 
SPS version 4.1, including maintenance releases 4.1a, 4.1b, and 4.1c, referred 
to collectively as SPS version 4.1 users. To make such an estimate was a 
challenge. Because there was no central source for identifying SPS users, the 
population was operationally undefined. The PMO had a central list of sites 
with SPS version 4.1. However, neither the PMO nor DoD Components had 
lists or access to lists of users at the sites with SPS version 4.1. Because there 
were no available lists of SPS version 4.1 users, there was no direct way of 
sampling those users. The absence of the baseline information on our target 
population substantially influenced the sample design used in the survey, as well 
as our analysis and projections. 

Sample Design. The sample design involved two basic steps. The first was to 
identify sites for statistical sampling. We organized sites into five groups, or 
strata. The strata were Army sites, Navy sites, Air Force sites, Defense 
Logistics Agency sites, and collectively all other Defense agency sites. 

To collect user surveys from a representative number of sites per stratum, we 
used two different sampling methods. For the 3 strata with 40 sites or fewer 
(Air Force, Defense Logistics Agency and other Defense agencies), we 
requested lists of users from all sites in a stratum. For the strata with lists for 
all sites, we drew a simple random sample of users from the combined lists for 
each stratum. For the 2 strata with more than 40 sites, we drew samples of 
28 Navy and 30 Army sites, respectively. We used a two-stage methodology 
for them, sampling sites within each stratum and then statistically sampling 
users within the selected sites. Table C-l summarizes the situation for each 
stratum. 


Table C-l. Sampling Methods for Stratum 


Stratum 

(Component) 

Total 

Sites 

Sample 

Sites 

User 

Census 

Sampling 

Method 

Army 

290 

30 

No 

Two Stage 

Navy 

182 

28 

No 

Two Stage 

Air Force 

19 

All 

Yes 

Simple Random 

Defense Logistics Agency 

9 

All 

Yes 

Simple Random 

Other Defense Agencies 

34 

All 

Yes 

Simple Random 


Measurement Issues. There were two main measurement issues in determining 
the number of SPS version 4.1 users throughout DoD: nonresponse to the 
survey by those sampled and the presence of nonuser names on the user lists. 
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There were two general types of nonusers. Persons selected from the lists for 
our sample could no longer be current license holders (because they were 
deceased, or no longer worked at the site from which the sample was drawn), 
or could be entered on the list more than once. As such, nonusers and duplicate 
entries were considered outside the scope of the survey. Across the 30 Army 
sample sites and 278 sample users, 2 persons were named twice on the Army 
list and 3 were identified as nonusers. 1 Among the 331 Navy users sampled, 
there were 7 nonusers. Of the 101 Air Force listed users, 1 was a nonuser, 
as were 2 of the 102 sampled from Defense Logistics Agency, and 1 of the 
101 from other Defense agencies. With these few exceptions, the remaining 
persons in the sample were considered to be current licensed users of some 
version of SPS. These were the basis of the Listed User population 
we projected. 

The nonresponse issue had two aspects that influenced determining the number 
of SPS version 4.1 users. First, some of the sampled users who were sent a 
survey did not respond to the survey, despite followup requests. The marginal 
response rate was nevertheless fairly high-205 of 278 for the Army, 217 of 331 
for the Navy, 68 of 101 for the Air Force, 70 of 102 for Defense Logistics 
Agency, and 81 of 101 for the other Defense agencies. 2 The proportion of 
nonrespondents varied somewhat from stratum to stratum from a low of about 
one-fifth to a high of about one-third. To calculate the number of Responding 
Users among all Listed Users, we have used the numbers of those who 
responded to the survey. Second, among those who did respond there are a 
number of persons who were using earlier versions of SPS. Since our aim was 
to survey users of SPS version 4.1, we projected both the number of 
Responding Users and the number of Responding Users who were SPS 
version 4.1 users. 3 At each of these points, because they were estimates, we 
have calculated not only the point estimate, but also the confidence interval, or 
uncertainty, of the estimate. 

Calculating the Number of SPS 4.1 Version Users. In calculating the number 
of SPS version 4.1 users, we have made the assumption that those not 
responding did so in a missing-at-random fashion and would have the same 


‘Among Army and Navy sites, if a list of users totaled 20 or fewer, we surveyed all 20, and any 
duplicates or out-of-scope persons counted directly against that total. Some persons were from 
sites with more than 20 users, in which case we added the next user on that site’s list in random 
selection sequence. For Air Force, Defense Logistics Agency, and the other Defense agencies 
pool of users, when one of the first 100 was out of scope, we took the next available user in 
random number sequence. The objective in each instance was to survey 20, or 100 users, 
respectively. 

2 These are marginal totals and part of a weighted sample design. They are reported for 
information only and cannot be directly projected to the sample as a whole without proper 
statistical weights attached. The sample sizes reported include out-of-scope persons. They, 
therefore, reflect all the listed names obtained and statistically represent the exhaustive lists for 
Air Force, Defense Logistics Agency, and other Defense agencies, as well as the sites 
statistically selected for Army and Navy. The total samples are the common baseline for all 
three population projections. 

3 We identified respondents as SPS 4.1 users if they answered "Yes" to Question 5, "I am 
currently using the 4.1 series of the SPS" or responded to Question 7, "Which maintenance 
release of SPS are you currently using?" with a particular release (4.l.a, 4.l.b, or 4.1.c). 
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proportion of SPS version 4.1 users as those who did respond. 4 We then used 
the ratio of “Listed Users” to “Responding Users” within each stratum to inflate 
the number of SPS version 4.1 users to its extrapolated proportion of all “Listed 
Users.” At the first stage, the DoD-wide projections were based on a stratified 
sample design and have been calculated using a 99.65 percent confidence level, 
as explained in Appendix A. Table C-2 presents the DoD-level projections. 


Table C-2. DoD Level Projections 



Lower 

Point 

Upper 


Bound 

Estimate 

Bound 

Listed Users 

8,679 

8,867 

9,054 

Responding Users 

5,371 

6,046 

6,721 

SPS Version 4.1 Users Responding 

3,828 

4,362 

4,897 


As Table C-2 above shows, the confidence interval associated with the estimated 
number of SPS version 4.1 users who responded was much greater than for the 
number of “Listed Users.” Furthermore, these confidence intervals were 
probably understated because they did not include the variance associated with 
using population estimates for the Army and Navy computations. The 
extrapolations reported below do not reflect the uncertainty in our direct 
estimate, nor do they include the underlying statistical uncertainty associated 
with using estimates of the population totals for the Army and Navy strata and 
other variable factors. 

Based on the statistical projections above, we have calculated an approximate 
number of SPS version 4.1 users out of all the “Listed Users.” In doing so, we 
have calculated stratum by stratum the ratio of “Listed Users” to “Responding 
Users” and applied that ratio to the projected number of SPS version 4.1 users. 
For example, the point estimate of the number of “Army Listed Users” was 
3,599; the point estimate of the number of “Army Responding Users” was 
2,558. This gave an Army adjustment ratio of 3,599/2,558, which we used to 
extrapolate from the Army SPS version 4.1 user point estimate of 2,037. Using 
the Listed/Responding ratio, we extrapolated from the 2,037 that there are 
2,866 SPS version 4.1 users among the 3,599 “Army Listed Users.” Table C-3 
sets out the intermediate, stratum level results, and the corresponding piece of 
our calculated extrapolation of the number of SPS 4.1 users throughout DoD. 


4 The concept of ‘missing-at-random’ assumption for missing data is further discussed in 
Appendix B. 
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Table C-3. SPS Version 4.1 Users 


Stratum 

Listed 

Users 

Responding 

Users 

SPS 4.1 
Version 
Users 

Responding 

SPS 4.1 
Version 
Users 
DoD-Wide 

Army 

3,599 

2,558 

2,037 

2,866 

Navy 

3,649 

2,312 

1,585 

2,502 

Air Force 

741 

504 

385 

566 

Defense Logistics Agency 

357 

250 

68 

97 

Other Defense Agencies 

522 

423 

287 

354 


Using the extrapolations reported in Table C-3, the total number of SPS 
version 4.1 DoD-wide users was 6,385. 5 We have not computed a confidence 
interval for this total because the absence of a well-defined user population has 
required a number of assumptions and approximations in calculating the 6,385. 
When reading the answer percents in Appendix B, the reader should keep in 
mind that the population figure of 6,385 is an approximation with a substantial 
unquantified uncertainty associated with it. 


5 The total number of SPS 4.1 users throughout DoD is based on the sum of the stratum totals and 
is not based on the ratio of the total Listed Users to the total Responding Users. The design is 
stratified and its totals must be calculated by summing the strata to take into account the 
different proportions of SPS 4.1 users among those responding in each sample. 


47 




Appendix D. Report Distribution 


Office of the Secretary of Defense 

Under Secretary of Defense (Comptroller) 

Deputy Chief Financial Officer 

Under Secretary of Defense (Acquisition, Technology and Logistics) 

Assistant Secretary of Defense (Command, Control, Communications, and Intelligence) 
Deputy Assistant Secretary of Defense (Deputy Chief Information Officer) 

Director, Defense Procurement 

Department of the Army 

Assistant Secretary of the Army (Acquisition, Logistics, and Technology) 

Auditor General, Department of the Army 

Department of the Navy 

Assistant Secretary of the Navy (Research, Development, and Acquisition) 

Naval Inspector General 

Auditor General, Department of the Navy 

Department of the Air Force 

Assistant Secretary of the Air Force (Financial Management and Comptroller) 

Auditor General, Department of the Air Force 
Deputy Assistant Secretary of the Air Force (Contracting) 


Other Defense Organizations 

Defense Contract Management Agency 
Defense Finance and Accounting Service 
Defense Information Systems Agency 
Defense Logistics Agency 

Non-Defense Federal Organization 

Office of Management and Budget 
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Congressional Committees and Subcommittees, Chairman and 
Ranking Minority Member 

Senate Committee on Appropriations 

Senate Subcommittee on Defense, Committee on Appropriations 
Senate Committee on Armed Services 
Senate Committee on Governmental Affairs 
House Committee on Appropriations 

House Subcommittee on Defense, Committee on Appropriations 

House Committee on Armed Services 

House Committee on Budget 

House Committee on Government Reform 

House Subcommittee on Government Efficiency, Financial Management, and 
Intergovernmental Relations, Committee on Government Reform 
House Subcommittee on National Security, Veterans Affairs, and International 
Relations, Committee on Government Reform 
House Subcommittee on Technology and Procurement Policy, Committee on 
Government Reform 
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Director, Defense Procurement 
Comments 




OFFICE OF THE UNDER SECRETARY OF DEFENSE 

3000 DEFENSE PENTAGON 
WASHINGTON DC 20301-3000 


FEBRUARY 6, 2001 


MEMORANDUM FOR THE INSPECTOR GENERAL, DEPARTMENT OF DEFENSE 

THROUGH: Director, Acquisition Resources and Analysis,^, 

Subject: Draft Audit Report entitled "Standard Procurement 

System Use and User Satisfaction" Project No, 
D2000FG-0091, dated December 20, 2000. 


Generally, I concur with the report's recommendations and 
have attached specific responses to each. The SPS Program 
Manager, the Military Departments, and the Defense Agencies have 
already implemented or ordered business process and software 
changes that respond to most of the report's findings. 

There are some questions regarding the interpretation of 
the survey results that should be addressed before a final 
report is issued. For example, although the report states that 
more than 60% of SPS users preferred a different procurement 
system, the data in Tables 7 and C-l show that substantially 
more respondents favored SPS than their existing legacy systems 
and more respondents favored SPS than other systems. ^Another 
example is the statement that SPS has not contributed' 
substantially to paperless contracting. More than 72% of the 
respondents reported that SPS had contributed to paperless 
contracting objectives. 

I appreciate the opportunity to comment on the draft report 
and ask that you provide site data to the SPS Program Manager. 
The site data will provide the Program Manager, the Military 
Departments, and the Defense Agencies a better understanding of 
the Audit findings and permit focused management attention. 


Deidre A. Lee 

Director, Defense Procurement 


Attachment: 
As stated 
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Director, Dafana* Procurtoant conaantj 

DoDIG Draft Audit Raport da tad Dacambar 20, 2000 
Project No. D2000FG-0091 

Standard Procurement System Uaa and Dear Satiafaction 


General Comment 

The draft report does not provide sufficient information to 
determine whether the dissatisfaction expressed by some users 
stems from: (i) the functions the software performs or does not 
perform; (ii) training and operational guidance that do not 
match product capabilities; (iii) user reluctance to adapt to 
the new business processes and procedures implemented by the 
software; or (iv) some other cause. DoD's ability to implement 
corrective actions will be improved if the final report 
identifies or provides further insight into the causes of any 
user dissatisfaction. 

Specific Comments on DoDIG Recommendations 

A.l. Me racoaund that the Director, Defense Procurement, direct 
the Standard Procurement System Program Manager: 

a. Perform adequate teating to ensure that each future 
veraion has the functionality required to meet the needs of 
intended users prior to the release of each new version of the 
Standard Procurement System. 

Comment: Concur. 

A formal, multi-step, testing process is an integral part of the 
SPS acquisition strategy and has been in place since contract 
award to assure that the software meets contractual requirements 
and user needs. The contractor tests each software release to 
assure that the software is ready for tender to the Government. 
Following tender, each release undergoes Government acceptance 
testing to verify that the software satisfies contract 
requirements. The independent operational test community and 
the user community evaluate each major software release in the 
field to determine whether the software satisfies user needs. 

The user community also evaluates each minor software release to 
determine if the software meets validated joint user needs. The 
contractor is responsible for correcting all failures to meet 
contractual requirements. Potential enhancements suggested by 
the user test processes are referred to a structured 
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requirements process that assures user recommendations are 
considered- 


• At the Department/Agency level for applicability 
throughout the Department/Agency involved; 

• By an inter-agency requirements board for 
applicability DoD-wide; 

• By an SPS Council (SES, 0-6, and GS-15 level) that 
reviews each recommendation for compliance with law, 

FAR and DFARS requirements, and emerging DoD and 
Federal business processes; and, 

• By an SPS Senior Steering Group (the Director, Defense 
Procurement, the Military Department Senior 
Procurement Executives, and representatives of the DoD 
Chief Information Officer) to resolve any differences 
of opinion regarding a recommendation's applicability 
across DoD and approve any business process or DFARS 
changes that might be required to implement a 
recommendation. 

b. Develop quantifiable performance measures that gauge 
whether the Standard Procurement System; 

1. Meets the mission objectives. 

2. Increase* productivity of contracting personnel. 

3. Achieve* the goals of paperless contracting as 
envisioned. 

Comment: Partially concur. 

The SPS software is a commercial product that is being 
modularly enhanced to satisfy DoD requirements. Consistent with 
Public Law and DoD and Federal guidelines for the acquisition of 
commercial items, the SPS acquisition strategy explicitly 
accepted commercial product performance for the initial software 
release and subsequent requirements identify the functions DoD 
wants the software to perform without specifying how the 
software must perform those functions. 

1. The SPS Operational Requirements Document (ORD) 
establishes the target and objective requirements for satisfying 
mission needs. 
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2. A general productivity performance measure is not 
practicable because the software is deployed to contracting 
activities that have different supporting infrastructures that 
affect software performance. The lack of a common baseline 
would not permit establishing meaningful, DoD wide, productivity 
measures for the SPS software. SPS's real contribution to the 
acquisition system lies in its ability to exchange information, 
using standard data, with financial, logistics, and requirements 
generation systems. Performance measures for that criticality 
capability could be developed when the automated financial and 
logistics systems come on line. 

3. SPS paperless performance measures can be developed 
subsequent to development of firm information exchange 
requirements for the DoD "End-to-End process" (DR1D 41) and 
implementation of those requirements at the contracting sites. 

c. Xvaluate each new version of the Sta nd ard Procurement 
System against performance measure* before deploying a new 
veraion or maintenance release. 

Comment: Concur. 

The SPS evaluation process is described in the response to 
paragraph A.1.a. 

d. Determine corrective actions, when the performance 
measures indicate the Standard Procurement System is not meeting 
mission requirements, user requirements, and delivering intended 
benefits. 

Comment: Concur. 

The SPS corrective action process is described in the 
response to paragraph A.l.a 

A.2. Ke recommend that the Director, Defense Procurement, direct 
the DoD Components to: 

a. Coordinate with the Program Manager, training and the 
transition to the Standard Procurement System Program so that 
tha Standard Procurement System is available for use when 
personnel receive training classes. 

Comment: Partially concur. 

The SPS Program Manager conducts weekly reviews with the 
Military Departments and Defense Agencies to assure that 
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training needs are identified in advance of site installation 
and training courses scheduled as close to the site installation 
date as possible. The Departments/Agencies are responsible for 
identifying their respective site training needs. 

b. Provide refresher courses for individuals receiving 
training too far in advance of the transition to the Standard 
Procurement System Program. 

Comment: Partially concur. 

The Military Departments and Defense Agencies are 
responsible for assuring that their personnel are adeguately 
trained to perform assigned missions. However, the Director, 
Defense Procurement at the next SPS Steering Group meeting will 
remind the Departments/Agencies to determine whether any 
employees at their contracting sites require additional SPS 
training and to coordinate any training requirements and 
schedules with the SPS Program Manager. 

B. W* rsccoMnd that the Director, Defense Procurement: 

1. Require DoD Component*, prior to any future deployment 
of either • new version or maintenance release, to: 

a. Provide documentation showing that it ha* determined 
that the Standard Procurement System has the functionality to 
meet each requesting site's aiission. 

b. Determine the number of employees at each site who 
require the Standard Procurement System to perform their jobs. 

c. Provide support for the number of license* requested by 

site. 

Comment: Partially concur with recommendations B.l.a. through 

B. l.c. 

Recommendations B.l.a. through B.l.c. describe management 
actions that the Military Departments and Defense Agencies 
perform today. Those actions are tailored for individual 
Department/Agency organizational and mission related 
requirements. Requiring the Departments/Agencies to submit 
documentation to the Director, Defense Procurement adds 
administrative costs to the deployment process, delays the 
deployment of new or improved functionality needed to satisfy 
user needs, and is not consistent with organizational 
accountability and responsibility concepts. 
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d. Correct existing records of Standard Procurement System 
licenses or establish such records, and redistribute licenses 
when employees no longer require the Standard Procurement System 
to perform their jobs. 

Comment: Partially concur. 

SPS site licenses are not user specific. Therefore, there 
is no need to redistribute licenses within a site. The transfer 
of licenses among sites can be accomplished within the license 
agreement or through negotiations. 

2. Direct ths Standard Procurement System Program Manager 
to establish review procedures prior to any future deployment of 
either a new veraion or maintenance releese to: 

a. Validate that documentation provided by the DoD 
Components, daterminsd that the Standard Procurement System will 
meat each site's mission, and that deployment to the cite is 
appropriate. 

Comment: Do not concur 

The Military Departments and Defense Agencies are 
responsible for meeting their respective missions and 
determining whether SPS meets their respective missions. The 
SPS Program Manager is not in a position to validate a 
Department/Agency determination that a particular SPS software 
release meets a site's needs or validate a Department/Agency 
decision to deploy SPS at a site. 

b. Validate based on functionality of the approved Standard 
Procurement System that the DoD Components determined the actual 
number of employees at each site who require the Standard 
Procurement System to perform their jobs. 

Comment: Do not concur. 

Site personnel requirements are a fundamental 
Department/Agency management responsibility as is the 
determination of which employees should use SPS. The SPS 
Program Manager does not have the responsibility, authority, or 
accountability for those fundamental Department/Agency 
management decisions. 

3. Direct the Program Manager to purchase licenses for all 
future deployments only after validation: 
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a. That tha Standard Proeuramant System meat* the mission 
requirement of tha Sita. 

b. Of tha actual number of employees who require the 
Standard Procurement System to perform their job at that site. 

Comment: Partially concur with recommendations 3.a. and 3.b. 

Licenses are acquired only in the quantities requested by 
the Military Departments and Defense Agencies to support the 
sites the Departments/Agencies designate to receive a particular 
software release. 
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DEFENSE CONTRACT MANAGEMENT AGENCY 

6350 WALKER LANE, SUITE 300 
ALEXANDRIA, VA 22310-3241 


F EB 7 2001 


MEMORANDUM FOR THE ASSISTANT INSPECTOR GENERAL FOR AUDITING, DoD 
THE DIRECTOR OF DEFENSE PROCUREMENT 

SUBJECT: DoD 1G Draft Report, Standard Procurement System Use and User Satisfaction, 

December 20,2000, Project No D2000FG-009I 


Thank you for the opportunity to provide written comments to the Standard Procurement System 
(SPS) draft report. This report a^kiresses SPS operational use from a procurement user's perspective - the 
efficiency for the user. There is also another measurement to consider - the effectiveness of the system. 


The Department of Defense is working to resolve material weaknesses regarding accurate 
financial records for oontract payments, that have been addressed in DoD IG and General Accounting 
Office findings. The And fina l***, communi ties r^tahlithing standard syston* and shared 

data to achieve accurate records and accurate contract payments as well as accurate reporting. As a 
critical feeder system to the Department 1 * accounting and finance systems, SPS is an essential element for 
achieving improved integrity of Financial Management Processes and Systems within the Defense 
Department - an effective solution 

SPS is being implemented through an incremental development and deployment strategy and 
therefore not all functionality is available until foil operational capability is achieved. Over the past five 
years, SPS has stayed the course. We fundamentally agree that there are various aspects of the overall 
SPS program that require vigilance, close coordination between the program and component management 
offices, the software vendor, and the user community. This reason is why we concur with the intent of 
the majority of the recommendations. However, I must caution that some of the recommended solutions 
by the draft report are not consistent with DODD 5000.1 requirements for ACAT programs. We have 
learned many lessons from the initial deployment of SPS and have, as a result, changed our processes for 
requirements, testing, field validation, and deployment. 


If you have any questions regarding the attached comments highlighting our current processes in 
regards to your recommendations, please contact the SPS Program Manager, Mr Gary Thurston at (703) 
227-4525 or by Internet: gthurston@hq.dcma.mil. 



TIMOTHY P. MALISHENKO 
Major General, USAF 
Director 
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SUBJECT: DoD IG Draft Report, Standard Procurement System Use and User Satisfaction, 20 
Dec 2000, Project No. D2000FG-0091 __ 

Finding A: Standard Procurement System User Satisfaction. 

Despite the problems identified with SPS version 4.1, respondents stated that SPS has the 
potential of being a very effective and useful tool for handling DoD procurement functions. 
However, for SPS to achieve its potential, the PMO needs to ensure that: 

• future versions of SPS are adequately tested; and 

• performance measures that track whether SPS meets the mission objectives, delivers 
the intended benefits, and provides measurable improvements to mission 
performance are developed. 

DCMA Comments: 

Concur that SPS version must be adequately tested. Our current testing process begins at the 
SPS Joint Requirements Board (JRB). A Military Department or Defense Agency headquarters 
and field representative comprise the membership, and they approve all requirements in a joint 
deliberation process that conducts various levels of coordination with their user community. 
These approved requirements are placed on contract with the vendor. During development, 
there is interaction with the vendor and the JRB about design and implementation to meet the 
stated need. This two-way, iterative and participative process also includes close review of 
technical design consideration by the Technical Working Group, composed of Component 
members. We are utilizing a Joint testing process in that we bring in field users to participate in 
testing the product during the development phase prior to vendor delivery to the government for 
acceptance During this phase we work out problems in the product and problems in design or 
possibly the requirement. We believe this to be an effective test process. 

During the acceptance test, we use field personnel to assemble the government acceptance team 
who will conduct scenario/script testing for acceptance. Once a product meets acceptance 
criteria, we move into a field validation phase, to confirm that the product operates in the 
representative operational infrastructure, for final check. We have continued to use this process 
to provide the Military Departmenls/Defense Agencies full participation to determine if the 
version meets the mission for the intended procurement community. 

Performance measures are defined in the SPS Operational Requirements Document (ORD). The 
Key Performance Parameters (KPPs) describe the measures of effectiveness to meet the mission 
objective (fix the material weakness), deliver the intended benefit (eliminate legacy procurement 
systems), and to provide measurable improvement to mission performance (resolve unmatched 
disbursements). The Program Manager continues to test and report on meeting the KPPs as well 
as the independent test agent • the Joint Interoperability Test Command (JITC). During 2000, 
JITC conducted an Operational Assessment (OA) at 24 procurement offices using SPS Version 
4.1. In a September 2000 report on the results of the OA, the JITC concluded the following: 

• Many procurement specialists have not yet accommodated to the changes in the way 
procurement actions will be done in SPS in comparison to previous legacy processes 
and methodologies. 
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Dec 2000, Project No. D2000FG-009i ____ 

• Services/Agencics that have standardized implementation guidance and standard 
operating procedures (including workarounds and changes to procedures/ processes) 
have experienced a more effective and less traumatic transition to SPS. 

JITC recommendations are to improve program support to include providing more user feedback 
concerning evolving functionality, dissemination of lessons learned and workarounds, fielding 
complete functionality, and improving vendor Help Desk support. The Program Management 
Office (PMO) and contractor continue to participate in Component user conferences to provide 
more user feedback about current and future functionality as well as improvements to the Help 
Desk support. 

Recommendation A.l: We recommend that the Director, Defense Procurement direct the 
Standard Procurement System Program Manager to: 

a. Perform adequate testing to ensure that each future version has the functionality required to 
meet the needs of intended users prior to the release of each new version of the Standard 
Procurement System. 

b. Develop quantifiable performance measures that gauge whether the Standard Procurement 
System: 

1. Meets the mission objectives. 

2. Increases productivity of contracting personnel. 

3. Achieves the goals of paperless contracting as envisioned. 

c. Evaluate each new version of the Standard Procurement System against performance 
measures before deploying a new version or maintenance release. 

d. Determine corrective actions, when the performance measures indicate the Standard 
Procurement System is not meeting mission requirements, user requirements, and delivering 
intended benefits. 

DCMA Comments: 

DCMA concurs with the intent of this recommendation. Since SPS is a major acquisition 
program, the responsibility for these recommendations resides with the Assistant Secretary of 
Defense (Command, Control, Communications, and Intelligence) as the Milestone Decision 
Authority (MDA). The MDA is assigned the acquisition management responsibilities to oversee 
the program cost, schedule, and performance. The needs are defined in the Mission Needs 
Statement (MNS) with performance parameters documented in the ORD. A September 2000 
memorandum from the Deputy Secretary of Defense requires systems, interfaces, and processes 
to eliminate problem disbursements that continue to plague DoD business execution. This 
memo initiates implementation of the Procurement-Finance To-Be Model and reinforces that 
SPS is an effective solution to a documented Defense material weakness. 

Adequate testing of the software capability is a critical and integral part of the PMO and 
Component Management Office (CMO) efforts to ensure the adequacy of each SPS version. 
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Dec 2000, Project No. D2000FG-0091 _ 

The program office government test lab performs the function of acceptance testing for each 
software version, maintenance and service release against contract requirements. This function 
is accomplished with the support of the component organizations, that provide individual testers 
to perform the test scenarios and scripts. However, the test lab evaluates the delivered software 
version, maintenance, and service releases to ensure they are compliant with the contractual 
requirements as approved by the SPS JRB 

The contract for a given SPS version does not specify performance measures. This is the 
result of the acquisition strategy to acquire commercial-ofT-the-shelf-based capability with 
incremental enhancements. The audit report accurately captures the fact that the ORD provides 
parameters for system performance and capabilities through thresholds for data accuracy, data 
relevancy, data currency, edit checks, data integrity, and operational availability. The 
government testers evaluate data accuracy, data currency, edit checks, data integrity, etc. as 
stated in the ORD. However, the SPS program and requirements baseline were not established 
to achieve productivity enhancements with respect to the missions of the individual components, 
services, and agencies, but for the DoD as a whole. 

The SPS version 4.1 has the capability to perform paperless contracting. Procurement 
communities can generate electronic images of the contract and pass them electronically to a 
web server for search and retrieval by the finance communities or others who have been 
provided access. The full implementation of a paperless contracting environment is dependent 
upon the data exchange between procurement and logistics/fmance systems. Programs and 
efforts are underway to achieve the capability as envisioned in the Procurement-Finance To-Be 
Model. 


Should SPS not meet mission requirements nor deliver intended benefits, the MDA, as 
the oversight acquisition manager, is responsible for directing program changes. DoD Directive 
5000.1 and DoD Instruction 5000.2 contain language for conducting family-of-system reviews 
and mission area reviews. These reviews will perform compliance reviews for all systems and 
processes within the finance, accounting, and feeder system portfolio. 

Disposition: Action is considered completed. 


Recommendation A.2: We recommend that the Director, Defense Procurement, direct the DoD 
Components to: 

a. Coordinate with the Program Manager, training and the transition to the Standard 
Procurement System Program so that the Standard Procurement System is available for use 
when personnel receive training classes 

b. Provide refresher courses for individuals receiving training too far in advance of the 
transition to the Standard Procurement System Program. 

DCMA Comments: 

DCMA defers to the Director, Defense procurement for responding to this 
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SUBJECT: DoD IG Draft Report, Standard Procurement System Use and User Satisfaction, 20 
Dec 2000, Project No. D2000FG-0091 ___ 

recommendation. 


FINDING B: Use of the Standard Procurement System 

DoD Components should carefully consider decisions to deploy SPS to the remaining activities. 

.. .The DoD Components should then provide justifications to the PMO that: 

• SPS has the necessary functionality to meet the mission needs of the site, and 

• Provide support for the number of licenses required. 

DCMA Comments: 

The SPS program is structured to meet the mission needs of DoD and not a site. Those 
needs are documented in the MNS ■ retire legacy systems and resolve unmatched disbursements. 
Through incremental development and deployment, SPS is delivering increased functionality 
with each software version. An Overarching Integrated Product Team (OIPT) meeting, chaired 
by the MDA, was held in October 1998 that resulted in an Acquisition Decision Memorandum 
(ADM) being issued on October 29, 1998. This memo directed that Components must 
determine if the version functionality meets the intended procurement community needs and 
then request that the deployments be initiated. See DCMA comment to Finding A for the 
process utilized for such a determination. 

The PMO requests deployment planning information from the Components on a yearly 
basis. The Components submit these plans that define the site location and number of users to 
be installed and trained. The PMO Deployment Team conducts weekly meetings to review 
upcoming deployment plans and make revisions due to infrastructure survey results, operational 
site considerations, and product functionality capabilities. The delivery order for the installation 
and licenses are ordered by the contracting officer, usually two to four weeks out from the actual 
installation. 

Recommendation B.I: We recommend that the Director, Defense Procurement: 

1. Require DoD Components, prior to any future deployment of either a new version or 
maintenance release, to: 

a. Provide documentation showing that it has determined that the Standard Procurement 
System has the functionality to meet each requesting site’s mission. 

b. Determine the number of employees at each site who require Standard Procurement 
System to perform their jobs. 

c. Provide support for the number of licenses requested by site. 

d. Correct existing records of Standard Procurement System licenses or establish such 
records, and redistribute licenses when employees no longer require the Standard 
Procurement System to perform their jobs. 

DCMA Comments: 
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DCMA partially concurs with this recommendation. Each Component performs 
validation after the PMO accepts the software to validate that interfaces are operational, software 
works in the Component architecture and environment, and meet their business needs Upon 
doing such, the Component confirms their written requests to the PMO for a new installation and 
training. The Component specifies the site location and intended users. It is also a 
responsibility of the Component and site to maintain the distribution of licenses in accordance 
with the contract and license agreement. The partial concurrence is that one must also keep in 
mind that licenses are purchased on a site user size basis and redistribution is limited by the 
contractual terms and conditions and the existing license agreements. • 

Disposition: Action is considered complete. 

Recommendation B.2: We recommend that the Director, Defense Procurement: 

2. Direct the Standard Procurement System Program Manager to establish review procedures 
prior to any future deployment of either a new version or maintenance release to: 

a. Validate that documentation provided by the DoD Components, determined that the 
Standard Procurement System will meet each site’s mission, and that deployment to the 
site is appropriate. 

b. Validate based on functionality of the approved Standard Procurement System that the 
DoD Components determined the actual number of employees at each site who require 
the Standard Procurement System to perform their jobs. 

DCMA Comments: 

DCMA does not concur with this recommendation. The PMO should not validate that 
the deployment of a SPS version will meet a site's mission. And the PMO has no business in 
validating the actual number of employees to receive SPS. These are Component 
responsibilities. The PMO does have a history of projected users at each procurement site and 
confirms if the Component request for site user licenses is reasonable 

Disposition: Action is considered complete. 


Recommendation B.3: Direct the Program Manager to purchase licenses for all future 
deployments only after validation: 

a. That the Standard Procurement System meets the mission requirement of the site, 

b. Of the actual number of employees who require the Standard Procurement System to 
perform their job at that site. 

DCMA Comments: 

DCMA does not concur with this recommendation. Again, the PMO should not validate 
that the deployment of a SPS version will meet a site's mission. And the PMO has no business 
in validating the actual number of employees to receive SPS. This is a Component 
responsibility. 

Additionally, The PMO does have a data base history of projected users at each 


Renumbered 
as Recom¬ 
mendation 
B.2. 

Deleted 


Renumbered 
as Recom¬ 
mendation 
B.3. 

Deleted 


Page 5 


63 




SUBJECT: DoD IG Draft Report, Standard Procurement System Use and User Satisfaction, 20 
Dec 2000, Project No. D2000FG-0091 _ 

procurement site and performs yearly data calls to update the data base and project program 
costs. The PMO does cross check the data base to verify if the Component request for site user 
licenses is reasonable. Licenses for siles are ordered in accordance with the contract line item 
structure. Costs for licenses are stair stepped depending upon the number of users: 1,2-10, II- 
24, 25-50,51-100, 101-250,251-500, and 501-1000. A 10% to 40% change in the number of 
actual users does not change the cost for the site license. Written guidance from the PMO to 
Components requires that Component requests site licensing based upon the number of users 
performing contract writing, i.e., civilian 1102 job series or equivalent military positions. 

Disposition: Action is considered complete. 
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